Automatic updates are coming to Drupal at the end of October! Long one of the most commonly requested features in the Drupal community, Drupal 7 and D8 will soon have an automatic updater that will allow Drupal installations to stay up-to-date more easily. How does Drupal's new auto updater work, and what do you need to know about it? In this Tag1 Team Talk, we dive into not only Drupal's new automatic updates feature itself but also its architecture, components, and roadmap, as well as why it's such an important part of Drupal's Core Strategic Initiatives.
Join moderator Preston So (Contributing Editor, Tag1 Consulting) and guests Lucas Hedding (Senior Architect and Data and Application Migration Expert, Tag1), Tim Lehnen (CTO, Drupal Association), Fabian Franz (Senior Technical Architect and Performance Lead, Tag1), and Michael Meyers (Managing Director, Tag1) for a deep dive into the nuts and bolts of Drupal's groundbreaking automatic updates feature, directly from the module maintainer, and the strategic initiative sponsors including the Drupal Association, MTech, Tag1, and the European Commission.
------------------------------------
Further reading
------------------------------------
Automatic Update - Module
https://www.drupal.org/project/automatic_updates
Automatic Updates - D7 and D8 Documentation Pages
https://www.drupal.org/docs/7/update/automatic-updates
https://www.drupal.org/docs/8/update/automatic-updates
Automatic Updates - Issue Queue
https://drupal.org/project/issues/automatic_updates
To provide your feedback on this first generation of the Automatic Updates module, create an issue in the Automatic Updates issue queue
Drupal Core Strategic Initiative
https://www.drupal.org/about/strategic-initiatives/automatic-updates
More information about the PSA.json feed can be found in this issue: https://www.drupal.org/project/drupalorg/issues/3068539
Drupal.org uses a package hashing and signing architecture based on the BSD Signify project
https://www.openbsd.org/papers/bsdcan-signify.html
Drupal contributors created a php implementation of signify: https://github.com/drupal/php-signify
Sponsors & Supporting organizations:
Drupal Association, Funding & Direction http://association.drupal.org
European Commission, Funding https://ec.europa.eu/
Tag1 Consulting, Development http://tag1.com
MTech, Development https://www.mtech-llc.com/
Text Transcript
-[Preston So] - Hello, and welcome once again to the Tag Team Talks series. Today I'm joined by several guests from all around the world and talking about a subject that is very near and dear to my heart, I'm very excited to hear about this topic, Drupal's new automatic update feature. This is part of a Drupal core strategic initiative happening as part of the Drupal core roadmap. And today we're gonna talk with the maintainer of the project, as well as several special guests, about what it is, why it's important, and how you can get started with it right away and help us report any bugs or any issues you might encounter. But before we get started I just want to remind everyone who's joining us today, don't forget to check out our previous Tag Team Talks at tag1.com/tagteamtalks. We've done several amazing webinars and sessions with guests from all around the world about realtime collaborative editing, rich text editors, and of course, our work, working with a top 50 Fortune company in all of these issues. If you do like this session, if you enjoyed this conversation today, please remember to upvote, subscribe, and share it with all your friends and colleagues at your team. So, first thing I wanna do today is introduce myself. My name is Preston So. I'm a contributing editor at Tag1 Consulting, moderator here at Tag Team Talks. Joined by my friend Michael Meyers, managing director of Tag1, located here in New York City. Both of us are separate places here in New York. We're joined today by Lucas Hedding from Leon, Nicaragua, the senior architect and data and application migration expert at Tag1. Lucas is one of the top 20 most active contributors to Drupal 8. He's also the Drupal core Migrate subsystem maintainer, a core contribution mentor, a drupal.org project application reviewer, and the maintainer for the thing we're talking about today, Automatic Updates. We're also joined today by Fabian Franz in Switzerland, senior technical architect and performance lead at Tag1. Fabian is one of the five Drupal 7 core branch maintainers. Here's also one of the top 50 contributors to Drupal 8 and maintainer for several Drupal 8 core subsystems, including BigPipe, Dynamic Page Cache, and Theme API, all very important systems in Drupal. We're also joined today by Tim Lehnen, located in Portland, Oregon, the CTO at the Drupal Association. Tim has been leading the Drupal Association engineering team for a number of years now, it's a great success, overseeing amazing initiatives to support Drupal development in the community. This includes the contribution credit system, the recent move to GitLab, and he's taken point on managing the relationship with the European Commission as well as they sponsor this wonderful initiative here. So, just to kick things off I wanna turn over the mic to Mike Meyers to talk briefly about Tag1 and why it is that we're talking about Automatic Updates, which is, once again, one of the most incredible topics, I think, that we can talk about in this series.
-[Michael Meyers]- Awesome. Thanks, Preston. Happy to be here. Tag1 is the second all-time leading contributor to Drupal. We have the largest concentration of core maintainers, and our team maintains over a dozen core systems like User and Views. We do a lot of our business based on Drupal, and auto uploader is key to keeping your sites up to date. So I'm excited to hear how this great new feature is gonna help end users, agency partners, platform companies, better manage their Drupal sites.
-[Preston]- Absolutely. And automatic update, this whole notion of auto update is, it's been something that's in the back burner of Drupal for a long time. It's also part of a very interesting aspect of the Drupal community which I think is very important to call out, which are the 10 or so currently ongoing Drupal core strategic initiatives. Before we jump into the technology itself let's just step back for a bird's eye view for a moment. What is a strategic initiative in the Drupal community, and what are some of the past and current ones? How did this whole initiative come to be?
-[Tim Lehnen]- I think that's a great question, and I'd be happy to speak to it a little bit. This is Tim with the DA. The strategic initiatives for Drupal are laid out on maybe an annual basis, maybe a little bit longer than that in terms of horizon, frequently in Dries' keynotes at DrupalCon, where he talks about key priorities for the Drupal project that he's discussed together with the whole core maintainer team and prioritized and settled on as the major efforts that he wants to rally the community around, and certainly the core developers around, in terms of moving the Drupal project forward. In the past, these have been things like adding Composer support to the Drupal project along with the release of Drupal 8. Things like updating the Migrate system, things like providing the API-first features in Drupal, and a number of things like that. And so, it was, I think in 2017 or 2018 when Dries first called attention to the need for automatic updates as a sort of foundational feature for Drupal, pointing out that this is something that a lot of commercial and proprietary software already does, and does well, and even some other open source projects are doing it. And it really makes a big difference to the total cost of ownership of people who run Drupal sites. [Preston]- Let's talk a little bit more about that angle, Tim. I think there's a lot of folks on this session here today who are really interested in learning more about what you just mentioned, the TCO, total cost of ownership. What exactly is the benefit of auto update to these small and medium-sized businesses? You mentioned agencies as well. What sorts of use cases are we supporting with this?
-[Tim]- Yeah, it's a great question. Let me paint a picture first for what happens right now around updates, and particularly the time period that's, I think, the most time intensive and critical for most people who maintain sites, whether they're agencies maintaining on behalf of clients or the end users themselves. So, that's security updates. Security updates are, of course, something that are very important, something that people need to keep up with as quickly as they can. And so there's sort of this culture and community within the Drupal community of everybody getting together during a core security release window on Wednesdays, hanging out in Slack, talking to each other, and waiting for that security release to drop. Now, while we have these security release windows, sometimes they come out right on time, sometimes they take several hours, but that window is sort of U.S.-centric because that's where a lot of Security Team members are. And so what that means is teams all over the world are up on high alert whatever time it might be locally waiting for this critically important patch to drop so they can update sites and make sure they're secure. And so, this is something that takes a lot of time that people can't really plan for, so much. They have to be there in that window, and they don't really have much of a choice. And so this is a situation that can be expensive in terms of just keeping your developers on call, paying overtime if you're outside of the timezone that these windows happen, all those sorts of things. And there's a lot of anxiety around them. That was one of the first elements that we wanted to address is making the security update process in particular easier, something you don't have to be so worried about, and then expanding on that to other use cases.
-[Fabian]- One question I had because it's pretty fascinating for me is for years, the Drupal community has said, we need auto updates, we need auto updates, we need auto updates. And it was always like, hey, but it's not possible in all. So, how come this direction change that now we are thinking it's possible finally?
-[Tim]- That's a really good question. I think there's a few factors that all came into play. And I think Lucas can speak to this a little bit more than I can. But just from an overall view, I think just various things came together. The architecture of Drupal 8 had changed, starting to do things like support Composer workflows, but also incorporating other elements like those, Symfony as a core element, all those kinds of things. But also we began observing just in the open source world other people doing this. So, for example, WordPress has an auto update system. Some other projects have an auto update system. So I think people have been thinking about how we're gonna architect something that would work in a Drupal context, for quite a while. Lucas, do you wanna maybe speak to what came together to make it more possible to consider? I think we lost your audio, Lucas.
- [Fabian] Lucas, you're on mute. Can you hear us? [Lucas Hedding]- Hello, can you hear me?
- [Fabian] Yes, much better. [Lucas]- Ah great. I think it's a bit of a myth that we had bought into. We had said to ourselves, we can't do this, and we said it to ourselves long enough that we had convinced ourselves. But then as we started seeing the competition in WordPress and others doing in-place updates or auto updates of some form we said, well, let's see if we can solve this tough problem. And we're close. We've spent a lot of time architecting this. We're still not 100% there, but I think we can see the light at the end of the tunnel. I think it's gonna be a pretty good success as we get to the finish of this first part of the initiative.
-[Fabian]- Now I'm really curious. How did you split up this mega task of getting even started with this tough problem?
- [Lucas]- Well, I mean... Let's see, we did a lot of architectural thinking, a lot of discussions at various camps and DrupalCons.
-[Tim]- Yeah, I think one of the... It was a confluence of a few factors that came together and made it work well. We had a meeting of minds between some key contributors in the core team at Midwest Drupal Summit in 2018 and did a lot of architectural work, and at the same time, with these conversations with the European Commission had started, and they were talking about how Drupal was a critical part of their infrastructure and they wanted to find something to support. And so we put two and two together and said, hey, we can do this. But yeah, scope is always a problem. It's a huge project. And so, what we said was, okay, well, if we wanna do some sort of phased scenario, what's the most important thing to start with, what has the biggest impact on people, and what simplifies the task a little bit? And that's where we came to security updates in particular as being the focus of this first phase because it both lets us focus on just patch releases that shouldn't be destructive in terms of what's being changed in those patches, and also fixes a critical need that a lot of the community has.
[Fabian]- That's really great. [Michael Meyers]- Maybe before we jump into the underlying architecture and how all this comes together, can you give us a background on how this works? Is this gonna update my production website in the background? Is this something I can do on my development environment and put through a CICD system? How does this work at a high level?
-[Lucas]- I think the answer is yes to all of those questions. It's supposedly flexible. At least that's how we're designing it. I think it's gonna be a little bit like the configuration management system in that there was an original intent for how we do this. And I think the original intent, at this point, is that you'd be a small site owner and you'd wanna in-place update on a live site, so definitely we're trying to make that as stable as possible. But if you've got a continuous integration system setup, there's nothing stopping you as a site owner from wiring into that. And then we'll iterate. This is an initiative that has multiple phases planned out already.
-[Fabian]- So it could be possible for me as... I mean, if I had 100 Drupal sites or something like that it could be possible for me to just get, instead of waiting for many, many hours on this Wednesday I would just get the package pushed to me, just as potential scenario, and then once the patch arrives then I can distribute it, test it, if my basic tests are working, distribute it automatically, basically.
-[Lucas]- Yeah, I mean... Well, some of the initial conversations I had was at MidCamp with folks from Pantheon and from, at the time, it was Drupal Commerce. They've renamed themselves to... I'm gonna butcher their name.
-[Tim]- Centarro is the name.
-[Lucas]- Centarro, right. So, we've got different folks in the room, and we're all just chatting about this idea of auto update, and everyone, Pantheon has its upstreams. There's nothing stopping any of these, even infrastructure owners, from certifying things and updating the URLs where you pull down files so that it's from a vetted source of thing where they've already done QA and added their secret sauce to this. I'm not saying that that's what's gonna happen because I don't know the future. But we're definitely trying to build this as flexibly as possibly so that even the hosting providers and your service providers can add their own secret sauce. Some of this might or might not play real well with what we're doing around the hashing and cryptographic signing of some of the resources. But at the very least, there's flexibility in the base architecture.
- [Preston]- Sorry, go ahead, Fabian.
- [Fabian]- I'm really curious as a Drupal 7 maintainer. Is this only for Drupal 8, or would Drupal 7, where we still have probably a million sites out there, also be? [Lucas]- Because of the source of our funding here we're in an interesting place where the backport policy for Drupal is you have to do it in Drupal 8. Eventually it'll be Drupal 9 and then we backport from there. But the funding and the interest from the European Commission has really been Drupal 7. So we're playing to two camps here. We've gotta fulfill our contracts with the European Commission and so all of the goodness of Drupal 8 is getting backported as we go along. And at this point, essentially everything that's in there for 8 has already been built out for 7.
- [Preston]- Very interesting. And I wanna get back to what you just mentioned, Lucas, about flexibility and some of the really interesting elements there. But first, we've established that this is gonna be for both Drupal 7 and Drupal 8 which is, I'm sure, music to everyone's ears listening right now. I'm curious though, because this is such an ambitious project as both you, and Tim, and Fabian, have all mentioned, where are we right now in the life cycle? I know, by the way, you have a full roadmap already outlined. There's gonna be a link, by the way, in the description of this video if you wanna check out the full roadmap for auto updates. But what's currently part of the initial outflow as of right now?
- [Lucas]- So, we did an alpha one, I don't know, three or four weeks ago. And we did that because we now have something of value. There's this concept that we're building into the auto updates called PSAs, or public safety alerts, announcements. So when these Drupalgeddon type events happen, when these majorly critical things are coming down the pipe, we can now have another channel, yet another channel, to alert even small site owners that their site is now out of date. We're doing yet another channel because we've already got the existing, hey, you've got security updates, hey, we've got Twitter, hey, we've got Slack. That only alerts and notifies folks if they're following the Twitter feed. And we already send out the emails about your site has updates available for it for everything already, so it's a little bit like the boy calling wolf every other week. So this is just another channel, but it's going to be used with great care. It's gonna be used once or twice a year. And then we'll be able to send out these alerts to folks, and we have that now for both 7 and 8.
-[Fabian]- Just to clarify for me, so, I'm one of those, probably not, but just imagine I'm one of those that always clicks away this message, oh, there's updates available. Yes, I know, I haven't updated this module in a while, or there's security updates, oh, but this module I can't, and so I'm missing a crucial core update. But instead of that message, this generic message, and many also disable Update module because it makes things slower at some parts, the thing is, instead of that, I would be getting the message, on Wednesday, the date, there's a critical security update, please pay attention, or something so that it's more direct communication to...
[Lucas]- Direct communication, it'll happen in the days preceding to the Wednesday as well. So, whenever the Security Team decides that, hey, this is important, they send out the emails a day or so in advance. This would also be part of their process. So the other part of what we've got built in in this alpha is what's called readiness checks, or preflight checks if you wanna put it another way.
-[Preston]- Sorry, go ahead.
-[Lucas]- The readiness checks just basically go out and say, are you ready? If you got an update tomorrow, or tonight, in 10 minutes would you be ready to update using this technology? And then we spent a lot of time thinking, well, how do we go about this? We even renamed it from preflight checks because with our non-English speaking community we thought it doesn't have as much meaning as a readiness check. Readiness checking goes and out sees, hey, is your site hacked? And I'll get into that in a second. Do you have enough disk space? If we were to apply an update, do you have two megabytes of disk space and you're gonna fail? There's about eight of these things that we decided. And some of them are errors, some of them are warnings. As an example, we'll warn you if you're not running cron on your site frequently enough. If you're not running it frequently enough we know that you're not gonna automatically update your site in the next couple hours. But that's a warning. We're not gonna block you. But if your site is mounted on a read-only hard drive for reasons of security, you can't really go out and update a read-only hard drive. That's just gonna fail. And that would be an error. So we've got this whole concept around readiness checks. It's a plugin, part of the Drupal 8 plugin system. Drupal 7 we've done something similar to that, not plugins, but the same business logic is in there. Can we update you quickly? And both of these things at this point are now available in the 7 and 8 alphas.
- [Fabian]- We can download this?
- [Lucas]- Yeah, right now.
- [Preston]- That's amazing. We've already spent a great deal of time talking about readiness checks, and I wanna get more into the features of the Automatic Updates module. But first I wanna move into a very interesting subject for our audience, because this has been a very uniquely challenging problem for Drupal for a very long time. How did you all manage to architect this into the right solution? And also one that could potentially be expanded out to other PHP projects, like WordPress, for example.
- [Lucas]- I don't know that I was involved in the early, early conversations. But since the beginning of 2019 I've been involved. Tim, you wanna bring up some of the early conversations?
- [Tim]- I can get into some of the early architectural discussions. I think the concern has to do... As Lucas was saying earlier, we have this myth that Drupal use cases and Drupal sites were so varied and so complex that that meant there wasn't a good way to have a set of standards for doing an auto update process. And we just really started interrogating those things in the last two years or so and said, well, is that really true? And what it came down to be is that the concerns were all about confidence. They were all about that ability to understand, hey, can we really be confident that these updates will apply cleanly, that they're not gonna break folks' sites, what do we do if we miss something and it does break folks' sites. So we spent a lot of this time saying, okay, what kind of architecture can we create to give us the maximum confidence so that whether you're running this as an attended or even unattended auto update you'll be able to feel like it's not gonna break your site. And so, that meant that we needed to architect these readiness checks, we needed to talk about something that's coming in the second phase which is this notion of having an A/B version of your codebase so that you can run an update, but if something fails flip back to the known good version, almost like a bootloader concept is what we were inspired by. Then we had thought about, okay, well, what else is involved? Well, we need to make sure that we can generate something that is the equivalent of a patch. Not literally the patch that's just been released but a quasi-patch, as Lucas termed it, that will apply all these changes cleanly to sites. And then finally we need to give people confidence in what's being delivered. So that means having a good way to secure the package delivery system that comes from drupal.org, ensure that it's signed and verifiable so that people know that what they're installing is in fact coming from a trusted source. All of these concerns are about confidence and integrity of the system, and so that's kind of what we built into it. And then once we realized, hey, you know, there's ways to solve each of these problems, there's ways to solve confidence that the update will apply before you do it, there's ways to solve confidence in being able to roll back if you need it, there's ways to solve the confidence in the trusted source of the package, then we realized it's almost significantly simpler. If you assume that we've solved those problems we're just applying patches and a couple processes to make sure those things apply clearly. And from there we got into the real weeds of the architecture which I think Lucas could probably speak to more. [Lucas]- It's interesting that you say it's simple because the patch that we've built right now for this service to auto update your site, it's a service. Half of this patch is testing. Actually, probably more of the patch is testing. It's like 500 lines, 400 lines of code, to update your site. Seriously, it's a really small amount of code to grab an artifact off of drupal.org, download it, and overlay that, I'm calling it a quasi-patch because it's not a full patch, it's not a regular patch. It's all the files that would be touched in a version between 8.73 and 8.74, as an example. If there's 10 files, we grab all those 10 files, put them in this patch. That way we don't have to worry about git apply, or patchutils, or anything on the server. We just have to deal with what's already built into PHP, copy and paste. And we do the copy and paste, and we do all of this... I'm gonna credit Nate Lampton here. His vision here was do it all in the same HTTP request. If you do it all in the same HTTP request and you roll back in the same HTTP request, the site can't actually go down. Well, knock on wood. It still probably could in very rare edge cases. It's about, I don't know, 200 or 300, 400, 500 lines of code, depending on if you start counting some of our tests in there as well.
-[Preston]- Very interesting. I think the way that you've solved this problem with obviously these quasi-patches and just the thinking that's gone into this really reflects, I think, the amazing commitment that you all have shown to making this a really successful feature. I wanna dig into some of the features now. We've already delved a little bit into some of the public safety messaging and the readiness checks. But just for our readers, and our listeners, and all the folks in the audience today, I wanna define just very clearly what those three major components of auto updates are. The first is public safety messaging, which we talked about briefly already. The second is readiness checks, or preflight checks, which really indicate when a site is ready to go. And then finally, I think the most compelling of all of these, which are the actual in-place updates that occur. What can we learn... Lucas, you mentioned earlier about Drupalgeddon and some of the ways in which, for example, every time I get a security advisory tweet or see that pop up, sometimes just the urgency of it doesn't really register with me, or things of that nature. I know Fabian also mentioned that as well. So I'm curious what sorts of other aspects of public safety messaging did your team consider during this whole effort? And what other components are involved in that?
-[Lucas]- Well, I really wish there was more folks on this call because it's not just the folks here on this call that have really contributed. It's been a whole team in the auto updates channel in Slack, and folks at different camps and conferences. So, what have we... I mean, there is other thoughts around just doing figuring out what is on the site already, and calling home to drupal.org and sending information. But there's security and privacy concerns around that, and so for the scope of what we're doing right now we're trying to keep it simple for messaging.
- [Tim]- Yeah, I would add to that, basically, we coordinated very closely with the Security Team, naturally, because this first focus was on the security side of Automatic Updates. And so, the main thing that we did was say, look, they already have their security advisory content type architecture on drupal.org, which is the basis of how the regular, standard, canonical announcements roll out. So what we asked ourselves was, okay, what are the requirements to changing that to provide a feed that could be consumed, a JSON feed that could be consumed by this auto update system, and then how do we put policy restrictions in place to say, hey, look, this is gonna be a more prominent message appearing in people's admin interface, but we don't want to cry wolf, as people have said, we don't want to overuse it, so what controls do we put in place. And mostly that will be at the discretion of the Security Team to say when is it worthwhile to put out this message about those additional updates.
- [Preston]- I think the interesting thing here is that we're really trying to reduce the burden on the user through this JSON feed, through the way in which we're gonna be able to get this alert directly to the audience. Now, just to move into more of the way that the user experience, or the way that the site owner experience is really improving here. These readiness checks, these preflight checks, that run on every six hours, or via cron jobs, how do they really play into this whole feature and this whole notion of automatic updates?
- [Tim]- I was just gonna add before Lucas runs into the architecture. From my two cents, what Lucas and others have architected is really great because it's an extensible system. New kinds of readiness checks can be created as time goes on. And what's been developed so far has been the result of core contributors, and the Security Team, and just other general contributors, coming together and saying, okay, what are those scenarios and use cases that are gonna be most important to determining whether or not an end user can actually run this auto update process? What's gonna block them from taking advantage of this new system that will make things easier? And so building this in so these currently eight or so different categories of readiness checks run on a regular basis and then provide their results reports so the site owner knows, oh, hey, if I fix this read-only directory, or if I fix whatever these warnings are or errors are in advance of this next round of updates going on, it's gonna be much more of a breeze for me. I just think those elements are gonna be really valuable. They're already valuable. They're already some things that I think people should be taking advantage of if they wanna try out the alpha. But I think the underlying message is they're only gonna become more valuable. As people test the alpha and as we move into the first phase stable release and things like that, we're only gonna learn more from contact with the world about other things that we could be checking, other things that we could be creating warnings, or errors, or messages for in order to help people be ready to take full advantage of the system.
- [Fabian]- Quick question to the readiness checks, because really interesting. As I've understood, it's plugin based?
- [Lucas]- It's plugin based indeed.
- [Fabian]- So as the site owner or, for example, a hosting provider, I could add my own readiness checks, as a site owner I could add my own readiness checks so if I need something more, like for example running some very basic tests or something that my homepage is still reachable, or whatever I wanted, I could just do that with a plugin.
- [Lucas]- Yeah, you could. We tried to come up with the basic ones that are very reusable. Most of these readiness checks are somewhere in the range, of real code, of about five to 10 lines. Can we write to a file on your hard drive? You know, I mean, that doesn't take that long to figure out that you can't do that. It's really simple. You tag a plugin with the right tags, look at an example, and now as an agency or even as a large company, if you have some specific rules around, hey, we need to make sure that the moon is in the third phase right now, you can do that. There's nothing stopping you.
- [Tim]- I would add...
- [Fabian]- That's really cool. And I have a question. If I have hacked core, and as a core maintainer I sometimes have to do that, can I still use the auto updates in that? I mean, is it preventing me from that, especially if a file is updated that I never hacked?
- [Lucas]- We're not blocking you from updating core, or any contrib module, if you have a file that's modified. Remember we're using a quasi-patch approach here. So, if you are about to update something... For the readiness check, if you've got a modified, patched file in core, we will warn you, saying there's some uncertainty here. Because what happens if a patch, a quasi-patch between 8-dot-something and 8-dot-something else comes out tomorrow? Well, you can look at what's been modified. If it's in Book module and you don't even use the Book module on your site, or the Book module hasn't been modified in six years or something... If you can look at it and you say, well, I've patched this thing and it's rarely patched, or there's no security issues in it that have come out for many years then you have greater confidence that you'd probably be okay. That said, when we go to apply that update we're gonna take that specific one and we're gonna say, no, no, no, no, you can't override a file if it's been modified.
- [Tim]- So any of your unchanged files, as long as you check from the warnings and you think it's otherwise okay, any of your files that are unchanged from what the architecture has, when it does its comparison, then yeah, those can still apply.
- [Lucas]- I wanna go into a little bit about this how are we figuring out if files have changed. Because I think that's a really interesting component to all of this, one that I wish I had had more... I wish I could say that I had more thought into it, but it's really the community. There's this whole concept, though, that makes sense after it was explained to me the first time about doing a file hash. Just do an MD5 sum, well, not an MD5 sum. A SHA-256 or a SHA-512 sum. We do that of every single file in all of core, all of contrib. The Drupal Association has really helped us out a lot here. They're gonna provide an official feed for hashes for all of these things. This is the type of stuff that if you look at the Linux distributions out there in the world they've been doing this for a super long time. If you try to download Composer, even, there's a SHA sum there. And you grab that SHA sum and you're supposed to do SHA sum to compare and to make sure that what you just downloaded is actually what was provided on the website. That's the type of technology we're building into. We're doing it not only to see is your site been modified, we're also doing it to have greater certainty assurance that these quasi-patch archives that we're downloading haven't been hacked or modified by a man-in-the-middle attack, or some other attack. And we'll link this if you're really wanting to look into it. There's a whole project that the Drupal community has now provided to the world on GitHub called PHP Signify. And it's using chain signatures. We've been going through it, trying to figure out... But you know what it is? It's following a pattern that BSD is using. BSD Linux has used the same thing. We're just standing on their shoulders. The current concerns about, well, did you write this hashing and verification system yourself? No, we didn't. And we also have some super, super smart people in the community contribute to this at the Midwest Developer Summit this summer.
- [Preston]- Very interesting, and I think this just indicates the amount of thought that went into this. And the notion of actually introducing your own readiness checks is something very interesting. We mentioned what phase the moon is in. I can see this being useful if you have, for example, a decoupled Drupal implementation and you have to wait on other code that you're working with to be ready before starting an automatic update, such as your decoupled application or frontend. So, now with these, the third feature we just named was in-place updates. Which, as you mentioned, Lucas, just now, you wanna be very secure. You wanna make sure that potentially these ZIPs that are being downloaded are not in any way affected by man-in-the-middle attacks. How do you look at that from the standpoint of the in-place update feature? What's going on in there in the actual Drupal site itself now that you've got those updates ready to go?
- [Lucas]- So, the mechanics of this small patch that we've got right now is that we download a ZIP file. The ZIP file has six files in it. Bootstrap.include is one of them, as in my example here. But another file that we have in here is this signature file that's called CSIG. It's using the same format that BSD Linux is using. The project that we're using is called PHP Signify. It's standing on the shoulders of libsodium which is baked into PHP as of 7.0 or 7.1. Someone will comment in the feed here and correct us. Which also has backwards compatibility shims all the way back to PHP 5.3. So if you're still on a Drupal 7 site and you're on an old version of PHP, you could still have this stuff work for you. It's doing a hash and it's using things like SHA to hash these files, compare the hash to a publicly signed and privately signed key pair of things. We're even talking about, not talking, we're doing, the Drupal Association is in the process of acquiring and installing an HSM so that we can do this in a very, very secure manner. And the HSM will then be taken offline. You can't get more secure than this. Is Drupal really a target? Someone, a conspiracy theory might say that it is. But even if it isn't a target, or even if it is, we're doing this in the most secure manner that we can.
- [Tim]- Sorry, just to add to that a little bit, I think it's really interesting and robust the way it's architected. Drupal.org delivers these hashed and signed files with the quasi-patch plus all of the appropriate signature files with the relevant metadata to verify all that. And then, again, as Lucas said, anyone using the auto update system, as long as they're on even a very old version of PHP with these shims in place, the PHP process can do the verification of the signatures, and the hashing, and all of that, and then we know that it's not an interrupted package that's been modified in flight, there wasn't a man-in-the-middle attack, or anything like that. And then on our side using an HSM we're able to do secure key rotations on a regular basis, we're able to validate, hey, you know, when was the last time this was checked, is it within a valid window of being signed, all this kind of stuff. So it's a really, really robust system. And I have to say, we're sort of, we have the advantage... It took us a while to get here, but we have the advantage of being able to look at the example of other projects, whether that's a BSD system that a lot of this is based on, but also look at, say, WordPress or other projects that have started implementing their own systems and say, these are the impressive things they've done but they may be missing this piece in terms of the signatures and man-in-the-middle protection, or maybe they're missing this other piece, and how do we get a lot of that right from the start. And part of this is only possible because Drupal has maintained its status as kind of a centralized project. Drupal.org is the home of most, but not all, code related to the Drupal project and that helps us be able to sign and deliver all the relevant packages. And with our move to GitLab, I think some other folks who aren't here are probably coming back as well.
- [Fabian]- And I'm very glad that you're doing that, especially with an HSM, because the recent Webmin hack showed how bad is this if you deliver hacked software to your users.
- [Preston]- Oh, absolutely. And I think a lot of our audience is very concerned about that just given the amount of risk that's possible here. So, I have one question for you both, Tim and Lucas. One of the things that I know a lot of our audience is curious about is, well, you have these archives, it makes sense. But what happens if we're using Composer? I use Composer. I've got Drupal 8.8. What exactly is planned for Composer? Or does it already work for sites that use Composer?
- [Lucas]- So, when you go to 8.8.0, when that gets launched in October, you will have Composer whether you know it or not. The Composer initiative has been really rolling and rocking quite quickly here in the last few months and a lot of things have come together such that when you upgrade to 8.8.8, 8.8.0, you're gonna have Composer just by installing and upgrading to 8.8. So that does open up the question. Because when this whole initiative of auto update started we said, well, we don't have any official Composer support in core. Remember, the funding is coming from European Commission. That funding will dry up here shortly. Let's focus on the needs of tarball. So we have spent a lot of time on tarball because of pragmatic reasons. But none of that means... None of that, though, means that we've not thought about Composer. And we have thought about it. And there's two or three different scenarios that we've considered with Composer. And the first two of them, you're fine. You're not gonna have an issue. Let me talk about those. The first one is you're on a site that's been using tarball since day one. And again, let me pause there. This is really an issue in Drupal 8. When we go back to Drupal 7, the number of sites that are using Composer are very minimal. So you can do it, I guess. I've never done it. Drupal 8 though, it is an issue. If you've been doing tarballs and you're still running before 8.8, let's just make it simple. You're not gonna have any issues at all. 8.8 comes around, you've upgraded to that because you're being a good boy or girl. Now you've got Composer. You're still okay because we're just overlaying files. The problem that comes in is when you start using the capabilities of Composer and you do a Composer require, and your Composer JSON and your Composer lock file start to get modified. In the case of that, you're gonna have some additional considerations. Again, if it's just an overlay of files you still will probably work in nine out of 10 times. Nothing that we've done will have an issue for your site. If you're wanting to patch, say, Token module, that's one of the more popular modules out there, and security release comes out for Token and it's been managed through Composer, we'll do that, that's not a problem. It's when you're dealing with a thing like... I'm gonna pick on Commerce. Commerce has a lot of vendor dependencies as well. If you want to update Commerce with this in phase one, and I wanna be clear, phase one, there are lots more plans in the future, you might have it a little bit more tricky. If it's just one file in Commerce, again, you still probably won't have a problem. But if it's a file... I've got this all written down in an issue on drupal.org. We'll link to it because this gets tricky. If it's a file that isn't in a vendor folder then you're fine. But if you start dealing with modified Composer JSON, vendors... But again, if think about the audience here, it's for folks that probably wouldn't have been doing composer update, composer require, up til now because they were on tarball. And so I think at least for version 1.0 of this module, we're gonna learn so much. So all that to say, Composer is thought of. There's more to come with it. And we'll make it better with time. We're gonna beef up that support.
-[Preston]- Absolutely. And I wanna ask just the other elephant in the room here that I know a lot of people have on their minds. Is this gonna be a part of Drupal core? Is all of this going into core? At what point do we see this being in Drupal core?
- [Lucas]- We've been working with the core contributors, and we even have Fabian on the call today. We've been working with core contributors from day one. This is an initiative out of the Drupal Association and this is not done in a vacuum. The simple answer is all of this will go into Drupal core. We're gonna incubate it in contrib. That said, some of the actual implementations of how we're doing this are not gonna go into core. We're not gonna do this whole overlaying of files thing on the live site, or at least not unless something changes, because there's too much risk there. Even though it's still very low risk, the core maintainers and release managers are still feeling that that's too much. So we've got plans for phase two, unfunded plans. I'm gonna put a request out there for funds. We need more funds for this A/B controller where you've got a live site on A and over on B you've got the about to be updated site and we're gonna flip back and forth between A and B, and if B has a problem we switch it back to A. And then we just keep doing that. And as we do that we can also start doing things more with Composer perhaps, beefing up that support. The simple answer is all of this will go into core.
- [Fabian]- What's the roadmap for the public safety announcements? Because there doesn't seem that much to it besides that needing to be a secure channel that I could commit that to Drupal 7, kind of right now. So what are the plans from the 8 maintainers there when we could get this in?
- [Lucas]- We're still in discussions with you guys, Fabian. Because we wanna figure out when's the most ideal time to take these features over. We've really been focusing on the end of the year deadline. But if someone were to grab the code, roll a patch, and throw it up there, there's no stopping anyone from RTBCing that, reviewing, testing, and getting that into core now because it's pretty stable. I say that, but I also say we've gotta learn a lot. This whole thing, we need to learn more. More information needs to be gathered. I was hoping that we would have a little bit of a chance to have several hundred sites, thousand sites, with the contrib module installed with this on there, figure out the right wording for this messaging so that we don't annoy people to death, but we don't do it too infrequently either, and iterate and improve on it in contrib where the gates are a little bit lower, the stakes are a little bit lower. And then when it's ready, move it into core.
- [Fabian]- Yeah, I mean, you could always start with a simple experimental module. has worked very well. And just one real quick question to the Composer thing, just if I got this right. So, the problem with comments is, for example, that a security patch might itself depend on a vendor library update and then I've got problems because then I can't know what kind of inner library's in there. Is that where the problems go into?
- [Lucas]- Yeah. I really probably should link it, and I feel like I did a poor description. But yeah, like, some of our more recent Drupal 8 updates worked for with a far file and we've got a vendor file in here. If someone has run composer update at some point and they've already updated to a newer version, and in the midst of all of this they got a newer version of Symfony and some other components, it could get really messy. I think what we're hoping to do is just release this so that folks can give us that feedback, is this gonna blow up on me. There's gonna be a few folks that are gonna be like, I don't care. Just give me the updated site. And those are gonna be the risky folks. Those are the guys who go down the black diamond ski slopes without any protection and half kill themselves. We need those guys out there to do this because they're gonna report, and probably get upset, that they broke their site. But we need those people to give us feedback. But if they don't give us feedback, if we give the folks the flexibility to hang themselves and there's silence, well that also tells us a lot too. It says, well, maybe we can relax some of our... Relax some of what we're doing here, and even maybe promote some of these more risky adventures.
- [Fabian]- One quick question for Composer itself. Have you thought about, the idea came out originally two years ago or something like that, to just run Composer inside of that process with a virtual file system?
- [Lucas]- Composer right now takes too much memory. And on shared hosting where you're capped out at 512 megabytes, or 256 if you've got other processes, it becomes cumbersome. But a lot of the same folks that have been in these rooms have been thinking about this for a while. So, more to come on that. I think we've got some ideas to drop that in phase two as funding comes available for the community. We might have some more surprises in the wings.
- [Fabian] Sounds great.
- [Lucas]- Tim, you know more about the future. Do you wanna talk about where you see this going?
- [Tim]- Why don't we talk a little bit, yeah, just about phase two and what's going on, and just move into a discussion of the future. As we've said several times, the European Commission, because they rely on Drupal and they have a commitment to all the open source technologies that they're a part of as part of their FOSSA program, approached us before the beginning of the year and said that they wanted to set something up, and generously they've been supporting the work so far to do this phase one of the project. But as we said, that's gonna run out. Then in the next phase when we wanna have better Composer support and this A/B system so that it can be actually brought into core, those are really, I think, probably the two most major priorities, we're going to really need to figure out how we create a sustainable funding model for that. For folks who are out there who may be at large end user organizations who rely on Composer in your workflows but are also interested in the possibilities of Automatic Updates, it'd be great to hear from you. You can reach out to me at the Drupal Association. And that phase two we're gonna try and really plant a flag around that A/B controller and more robust Composer support and see who we can bring to the table to help us make that happen. It's really important to us that we move this forward. It's a huge priority for the project, of course, but it's also, it's not been an easy problem to solve. As people have mentioned, we've been talking about this in the Drupal community for probably five plus years and are only now beginning to make some real progress. But it's taken work. As we've alluded to before, there's a huge group of people involved. Staff on the Drupal Association side. People like Tag1 who've been working with us closely to actually do implementation, Lucas in particular. The Security Team, all the volunteers in the auto updates channel. Just a huge amount of people. The core maintainers have all been involved. And without funding, we might make a little bit of progress but I don't wanna go another five years without really delivering an even more robust system that we're capable of doing.
[Preston]- Yeah, I think this definitely shows the amount of support and backing from the community but also from these sponsors that have really contributed a lot to make this a success. Obviously we mentioned the Drupal Association, MTech, Tag1 Consulting, the European Commission and their Free and Open Source program. I know that you all are looking for more sponsors. Tim, just to be very clear, to reach out to you they can go to tim@association.drupal.org.
- [Tim]- That's right. You can reach me there directly.
- [Preston]- Absolutely. I just wanna mention briefly that this is a very important initiative for the future of Drupal and for the ability for site owners to really have people, or to have that governance and maintenance that they've been used to all along with so many of these projects. And so Tim, just to jump back into the notion of sponsorship and looking for contributions. Why should folks look at sponsoring this project? We've talked about obviously the benefits technically, but are there other benefits also to this?
- [Tim]- Yeah, I mean, in addition to the technical benefits and the benefits of just being a part of good citizens of an open source community and raising the tide for all ships, there's also, because this is a strategic initiative for core, there's a lot of visibility into this project. There's opportunity for marketing and awareness campaigns around your support of this initiative if you choose to get involved. Cross-promotional activities that can be done with the Drupal Association. There's also opportunities for recruiting some of the best talent in the Drupal community if your organization is looking to recruit some more key talent. A lot of the contributors who've been involved, whether on a volunteer basis or a contract basis with this initiative are some of the most talented folks in the whole community, and you could recruit those people to your team. And then also, this is something that gets talked about on stage during the keynote at DrupalCon. If, for example, we managed to get someone on board before DrupalCon Amsterdam, I'm sure we would be thrilled to announce that partnership with another organization. Moving forward into the next year during the keynote, during Dries' keynote, we're certainly gonna be talking about the progress on this initiative and all the organizations who've been involved in the first phase so far.
- [Preston]- Wonderful, and I'm looking forward to hearing a lot more about this at DrupalCon Amsterdam. Before we close things up here, though, I just wanna ask, are there any other things that we wanted to mention about where this project is headed, or any other things that the audience might be interested in?
- [Tim]- I would just reiterate something that we said before, which is, there's an alpha available now. It's been out for several weeks. So if you're in a position to try it out, to test it out and provide feedback, that would be wonderful. If you're interested in being involved in the initiative there's a lot of activity in the auto updates channel in Drupal Slack as well as in the drupal.org/projects/autoupdates so you can get involved there and help out. And yeah, certainly feel free to reach out to myself or others involved if you have questions.
- [Fabian]- Awesome work.
- [Preston]- Yes, this is fantastic.
-[Lucas]- I mentioned it before but this has been the work of many, many people, and we've only got five or six on this call which is a poor representation of the larger community. So my shout out to Drupal and the Drupal community.
-[Preston]- Absolutely. All righty, well, we have run out of time here at the Tag1 Team Talk. So, just to remind the audience here today, thanks again for joining. It was a really amazing conversation about Automatic Updates coming to Drupal. A really amazing thing. I'm looking forward to hearing about it in Amsterdam and Minneapolis. Just for your information, we post all of these, by the way, at tag1.com/tagteamtalks. All the links are gonna be available alongside this video and audio recording. If you like this talk, if you like what you heard today, please remember to upvote, subscribe, and share it with all your colleagues, your parents, your friends, your grandparents too. And as always, please, if you have any topics you wanna hear about, if Automatic Updates is interesting to you, you wanna bring Lucas and Tim back into the fray here, please write to us at tagteamtalks@tag1consulting.com. I wanna say a big thank you to Lucas, a big thank you to Tim, and to Fabian, and to Michael as well, for joining us today for yet another Tag Team Talk. Thank you all, and goodbye.