Transcript - Drupal Security and Drupal end of life, with Michael Hess

This is a transcript. For the video, see Drupal Security and Drupal end of life, with Michael Hess.

Michael Meyers: [00:00:00] Hello, and welcome to another Tag1 Team Talk episode, the podcast and blog of Tag1 Consulting. I'm Michael Meyers, the managing director of Tag1. And we've got another awesome show for you today. This is part two of our two part series with Michael Hess, who is one of the leaders of the Drupal security working group and the Drupal security team.

In today's episode. Michael's going to give us a rare inside look into what the security team does and how they work and operate.

[00:00:26] I want to shift gears a little bit.

We've talked a lot about the security team and, and what it is and how it works. And I, and you might know where this is going, and I want to talk about where it doesn't work, specifically end of life. You know, we've, we've had one instance thus far, you know, Drupal 6 reached end of life around six years ago.

You know, and Drupal 7 and 8, 8 reaching end of life in a few months, and 7 is reaching end of life and, you know, a year in a few months. You know, this is a really difficult, perhaps controversial to some people, you know, challenging topic. You know, so maybe real quick, not everybody might understand what end of life is, you know, just~~ ~~in a brief overview, what is end of life and, and what is the difference between, you know, a current version of Drupal and how it relates to the security team and, and what happens when it crosses that that EOL line.

[00:01:23] Michael Hess: ** **So end of life is exactly what it sounds like. The, the product or service is no longer going to be offered. You know, this happens for a variety of reasons. It's not unique to Drupal. It is a software phenomenon. If you're familiar with the software development life cycle, like, this is something that happens at some point.

Software, you know, reaches the end of its its existence. We don't run Windows 3.1 one anymore. And you know, I will start that with saying there are actually some instances where people need to run Windows 3.1, because they've got some proprietary piece of hardware that only works with Windows 3.1, but if you walked into a store tomorrow and bought a brand new computer and it came with Windows 3.1, you would be a little concerned.

[00:02:09] 1996 called they want to know where their, you know, 286 is.

[00:02:13] Michael Meyers: What is this little piece of plastic and what do I do with it? A floppy disc. I've never seen this before.

[00:02:10] Michael Hess: And so end of life is, you know, the organization that makes the software is saying we are no longer going to support or maintain this software.

[00:02:30] And, you know, this has been a discussion, you know, with Drupal 6, we went through this process, actually at DrupalCon New Orleans, there was a mock funeral for Drupal 6, which was hilarious. Go look up the mock funeral and, you know, traditional New Orleans style. Drupal 7 was actually scheduled to hit end of life this year, at the same time as Drupal 8.

[00:02:57] And we extended that by a year, mostly because of COVID and budget impacts and, and you know, what was going to happen to people's sites as a result of, of the, of the end of life. And the impact that might have had, maintaining old software is expensive. You know, when you think about Drupal 7, Drupal 7 has been out for a very long time - when Drupal 7 was released, the PHP ecosystem was vastly different than it is today.

[00:03:28] And so just running Drupal 7 on a modern PHP stack is not something that anybody, when Drupal 7 was originally built, had in mind. and so we are end of lifeing the product. That effectively means that the security team will no longer provide coverage for the product. If a security issue comes into the Drupal security team, the security team itself won't address that.

[00:03:50] I'm going to put a star on that and come back to that in a second. The, you know, the, the core maintainers will no longer be committing new features or bug fixes to Drupal 7. So when PHP, you know, 9 or 10 comes out, yes, it's a PHP, not Drupal 9 or 10 comes out. The core maintainers will not be fixing Drupal 7 or Drupal 6, or Drupal 5 to work with those new versions of, of PHP.

[00:04:20] Now having said that, you know, it is a fact that people are running large, complex, massive systems on Drupal and us end of lifeing - it is problematic for their business. And so, you know, I go back to the folks that are still running Windows 3.1, and, you know, I don't, I don't actually know of anybody running Windows 3.1.

[00:04:40] I do know of a large group of folks that run Windows 98 though. and there's a couple folks that are still running, you know, Windows 2016 server, which is end of lifed. And Microsoft actually will sell you services to continue to maintain that. And the cost of those services goes up as time goes on because you know, when there's a hundred thousand people buying that service, okay.

[00:05:05] There's, you know, we can pay a couple engineers. What if 5,000 people buying that service or 5,000 companies buying that service? Well, okay. The cost to pay those engineers is now significantly more because there's less people paying. A Drupal 6, did a long-term support vendor agreement, where we had vendors who worked with the security team, Tag1 was one of them or is one of them.

[00:05:28] There are still customers who are using Drupal 6 and who are paying one of the, I think three room, three vendors. It may be down to two, who are. It started at four, who are providing Drupal 6 support for these folks that cannot move on because they've got business processes that are tightly integrated and the cost to pay a vendor is worth it to them.

[00:05:52] We will be doing the same thing with Drupal 7. Tag1 is also a Drupal~~ ~~7 - we're calling it extended support vendor. And so as a company, you can go to one of these vendors and you can basically say, this is my site. This is what I have. And the vendors have a variety of services, everything from notification about releases to the vendor logging into, and actually patching our site for that, and everything in between.

[00:06:21] And so the, you know, the, there's an application to become a vendor, and you go through this process. And so when a report comes in, you know, the security team, we will coordinate with the vendors who may decide to fix the problem, or may decide not to fix the problem and make it public. And, you know, you might say, well, wait a minute, how do I know if they're going to fix the problem or not?

[00:06:43] I'm not paying them. How do I know if they're gonna fix the problem? Well, if one of their sites of a paying customer is impacted, they will likely fix the problem. So, you know, if a critical, highly critical mass exploitable vulnerability for Drupal 7 gets reported, it'll go to the extended vendors.

[00:06:59] And they will almost certainly fix that. If, you know, an edge case gets recorded or a stress case gets reported, where, you know, it affects 1% of 1% of Drupal sites and none of their customers sites are in that percentage, then they're not going to fix the problem, even if it's a critical issue. and so, you know, the, the security team was pretty successful in doing this with the Drupal 6 environment.

[00:07:22] And we've replicated that for Drupal 7.

[00:07:24] Michael Meyers: We're not, the, the extended support providers are not replacing the security team so much as augmenting them. Because one, many of the people, well, you have to be on the security team to provide this service. So you're just getting paid to do it as opposed to, you know, volunteering to do it, which is necessary because of all what you described.

[00:07:45] Right. It's, you know, the use case, you know, there's so many, so few people using it that if you want this, you should pay for it. But, you know, I, I think the one I really wanted to point out there was, you know, the security team is still there. You know, while you guys are not driving this process, you're involved in it.

[00:08:02] You're aware of it, you know? And so, you know, I, I don't know how active you are in it, because again, it's a black box to me and I'm not allowed to know these things. But, you know, it, it seems like there's some level of coordination, if not, just because a lot of the vulnerabilities that. Or like trickled down.

[00:08:23] So a vulnerability in 8 or 9 impacts, you know, 7 or 6, by that nature, you have to be involved. And if something is discovered in 6 or 7, it might trickle up. And so, you know, it's not like you guys walk away and you're like, whatever, you know, 7, you know, like, you know, it's just that you're not, you know, owning and driving, you know, the way that you normally do.

[00:8:48] Michael Hess: The first thing we do when we, when we get a report for an end of life version of Drupal is we validate that it does not exist in a supported version of Drupal. And so, you know, if a vulnerability comes in and it exists in 8 or 9, then we - it's our process and we work with the extended support vendors.

[00:09:07] But it really goes through our normal processes. If it's just a 7 only thing, you know, the extended support vendors really own that process. We will not publish a security advisory about it. The fixed code doesn't actually even go into the canonical Drupal repo. It will go into another repo or the patch gets posted to an issue like the way that happens is up to the vendors.

[00:09:30] But there's some of the tooling that we have that won't, you know, won't be there. So for example, if you want to, right now, if I do a security release for Drupal 7, we have a test bot that runs that test bot will not be available once Drupal 7 is end of life. Why? Because we're saving the money on the infrastructure to run the test bots.

[00:09:43] Michael Meyers: I think runs a lot. I don't think people, like, I really don't think people understand the scope of Drupal. I mean, that thing did something like 10 concurrent years of testing in a single year, like a decade of effort, you know, like you're talking about a massive, massive AWS bill, you know, that's just not sustainable.

[00:10:09] So that, you know, that has to be decommissioned.

[00:10:12] Michael Hess: And you know, we'll continue running tests for Drupal 8 and Drupal 9, but we can't, you know, we can't support running tests for every version of Drupal. Having said that the infrastructure behind that desk bot is available to the, you know, to the vendors and honestly, everything about it is open source.

[00:10:27] Anyhow. So anybody could recreate that test bot if they wanted to. The tests are in the canonical Drupal repo. But, you know, a lot of the, a lot of the, the procedures, the policies, the tooling goes away because in some instances, this is now to the vendors to handle that level of support. And so, you know, a vendor might have a contract with a, a major, you know, organization that says, look, this is what we need.

[00:10:52] And the vendor may agree to it. And the vendors, then that's a contractual business relationship between the vendor and that company, the security team isn't really involved. You know, we do remind them vendor is that for issues that do impact the supported version of code, which may not be core, could be, you know, a module that is in both, 7 or 8, 9 or 10 and 7 or 6, that, you know, our processes and procedures come into place

[00:11:22] before, if, if it impacts a currently supported version, but you know, in some ways there's a lot of, you know, the vendors have, have the authority to say, we're not going to fix this. We're going to make this issue public. That's within their call and they, you know, they have to all agree to do that.

[00:11:37] And the other thing we did is we put in rules about vendor participation. You know, we wanted to make sure that vendors who are part of this program are actively fixing issues. Because it is, you know, the vendors get to self-organize for a little bit, but we do have some rules in there that basically, you know, say, well, you can't become a vendor and then not fix anything.

[00:11:57] Like that's not cool because then all the other vendors are doing the work

[00:12:01]Michael Meyers: So yeah, so as far as you know, for long-term support for Drupal 6, you know, I think that, MyDropWizard, Tag1 like vendors came together and proved that this model can work really well. So I think that organizations that are on Drupal 7, and looking for extended support should rest assured that one is going to be available.

[00:12:21] And two it's going to be available for a really long period of time Drupal 6 extended support is still available. It's been around for, I think, Drupal 6 went end of life, six years ago now almost five years ago. We had a big enterprise, a billion dollar corporation reached out to us yesterday asking for Drupal 6 support.

[00:12:40] Like people are still contacting us and asking for Drupal 6 support and it's still available. So if you're on 7, you know, you should be able to run 7 for another seven years with Drupal extended support from vendors. I did a DrupalCon talk, with help from Michael. You can check it out at tag1.com/eol for end of life, which explains the whole model, how it works, what it will do.

[00:13:04] But since I have you here, Michael, I think one of the controversial things, and, you know, I have a strong opinion on this, but I want to get your perspective. Drupal 7 is by far the most popular release of Drupal ever, there are more sites running Drupal 7 than every other version of Drupal combined.

[00:13:23] And I think a lot of people are wondering why would you end of life, you know, the most like, you know, loved and popular version of Drupal.

[00:13:34] Michael Hess: Well, it's not my call. I'm not the one who's deciding to end of life it. Uh, it's actually a really complex question. You know, part of it is that Drupal 7 is running in many, many different environments. And this goes back to my, you know, like site persona thing from earlier, in that, you know, there is the persona of, I set up my site seven years ago, I keep it updated.

[00:14:03] Haven't touched it. It's not complicated, but I also don't want to take the time to upgrade. But you know, I've got a hundred nodes and seven modules and, you know, like it's my site, it works. And also the I've built my entire business around it. There's a hundred million lines of code code in this. It is my ERP, like, so, you know, those are kind of the two opposite ends of the spectrum here.

[00:14:25] At some point the, the person who set their set up seven years ago, like their host is going to upgrade them out of Drupal. Like the amount of work it takes to keep chasing PHP versions. And, you know, without breaking old code gets complex and that's a, you know, that's a lot of work. The, the, you know, group that's run their entire ERP on it.

[00:14:47] They're going to have somewhat of the same issues. If they're using PHP 5.3, please don't be using PHP 5.3, first of all. But if you're using PHP 5.3, but that's a lot of work, their software, you know, goes through this process. I think, I don't know the exact dates. I probably should have been aware of this.

[00:15:06] I think they're, I think Drupal 7 has been around for a very long, like 10 years, I want to say.

[00:15:16] Michael Meyers: Oh yeah. And the longer. It's been in development. I mean, it was in development for something like four years before it's official, I guess development life cycle was really long. So, you know, it existed at least three, maybe four years before it's released and it's been live for at least eight or nine years.

[00:15:33] Michael Hess: So it's actually the Drupal 7 came out in January of 2011. Okay. so the original end of life date was going to be 10 years after it came out. there's very few software packages that are supported for 10 years by any vendor. And we've added a year to that, you know, The PHP universe is moving towards what Drupal 8 and Drupal 9 look like.

[00:16:02] And so, you know, there are competitors to Drupal and I'm not going to say you should go use a competitor to Drupal, but you know, in the evaluation of what to do, some of that is reevaluating your use case. And is, you know, his Drupal 8 worth the migration, you know, I've, I've got a lot of sites that have said, you know, I built a site 10 years ago.

[00:16:25] I haven't changed it in six. Like I can make this a static site. There's not a good answer here because, you know, it's, it's hard. Where do we balance resources? You know, we have a finite number of, of people who developed Drupal at the, you know, working on core, where do we balance those resources on the current version of the software, which has, you know, all sorts of modern day technologies built into it or on the ten-year-old software.

[00:16:50] That's got a lot of use, And so, you know, one of the things that this came up with with your question is why we did extend this. You know, I know there were a lot of migrations that were planned to start, or were in progress at the time the pandemic started. And, you know, we, we, we basically said, okay, we know these are going to cancel.

[00:17:12] We know budgets are going to get tightened and let's extend this a year. Well, you know, we need to move, we need to move forward from Drupal 7. That's where the software development processes are going to. What's your opinion on it? You said you had a strong opinion.

[00:17:29] Michael Meyers: Yeah, I mean, yeah, check out my tag1.com/eol talk.

[00:17:35] I talk a lot about this and you know, One, I, I think the community is amazing. And I, I love the fact that the community said, you know, look, COVID the world's in a difficult place. We're going to give everybody yet another year, we've already extended. You know, I feel like the community has gone out of their way to try and do everything they can.

[00:17:55] And what people have to realize is that we're like in our community hat capacity, we're largely volunteers. And so, you know, you're talking about software, that's, you know, a decade old since release three or four years in development. So I've been working with Drupal 7 for 13 years. Like people don't want to be working on 13 year old software, like not in their free time and, and even paid, right?

[00:18:18] Like we, we excel and, you know, progress in our careers because we are researching and learning the newest things. And so. You can't rely on an open source community for support. And so I think that the model that we've put in place with this extended support program, is a great blend because the community shouldn't provide it, won't provide it.

[00:18:46] But you know, putting this extended support program into place gives companies options. And again, like to me, Drupal's about freeing people from their technology, like, like, you know, the site builder role, you know, like a layout builder, you know, the, the, you know, Drupal empowers non-technical people to control and manage their site.

[00:19:07] And it enables technical people to do so much more. So for me, Drupal’s all about being control of your technology. And what extended support does is it puts business in control of their technology. It says you can make a business decision to upgrade. It might be make your site static, make it, you know, archive it, go to another platform, or it could be pay an extended support provider to help you continue to run and maintain your site.

[00:19:34] You don't have, the 7 is a great platform, you know, I mean, you know, if it meets your needs, you know, and there's no business reason for you to upgrade, extended support is perfect. You know, like, I don't think you should be held hostage by your technology. And so, you know, I don't, you know, I love that you don't have to upgrade, you can continue to run it and that extended support is there for you to do it.

[00:20:01] Michael Hess: And it's not like we've got a kill switch in the software. We're on, you know, the end of life date, all of your Drupal 7 sites will magically just stop working. Like we don't do that. It's an open source platform. You can also choose to take the risk and say, okay, you know what? I haven't done a security update in three years.

[00:20:18] Since that last major one. I'm going to take that risk and you're going to have, you know, I agree with your initial reaction. They're like, Oh, please don't do that. But people are going to do that. That's the beauty of open source. You know, when we talk about where funding in this community comes from.

[00:20:31] If you go look at the, you know, drupal.org marketplace, which is ranking people by their contributions to the platform or to the, to the product Drupal or modules or holding events, or any other ways in which people can contribute to this platform, you know, of these, I see two vendors that do hosting and the rest built sites on clients.

[00:20:55] And I can't imagine that there's a lot of clients that are going to these vendors and saying, Hey, I want a Drupal 7 site, build me a Drupal 7 site. And I know it has all these wonderful APIs. I want you to build them from scratch and Drupal 7. like that's not happening. And so, you know, the people who are getting the, who are, who are actively working on projects on behalf of clients are building them and building the new ones on the new services. And if you haven't updated to Drupal 8, because you know, your concern here is, Oh, well, I'm going to update to Drupal later, I'm going to upgrade to Drupal 9. And then, you know, X number of years, I'm going to have to do this massive upgrade and start over again. You know, the upgrade process is no longer rebuild your site.

[00:21:39] You know, when you went from Drupal 5 to Drupal 6, Drupal 6 to Drupal 7, and it was a quite painful process, which in some cases, effectively necessarily required you to just start over. Like, you're going to start with installing Drupal. You're going to import your content. You're gonna go find your modules.

[00:21:56] You're gonna have to rewrite chunks of your custom code. You know, going from Drupal 8 to 9 can be as simple as just updating the code and running the migrations, depending on what code you're running and what you're doing in there, it is not necessarily rebuilding everything. And so, you know, that is one of the huge, huge benefits of being on Drupal 8, 9 or 10.

[00:22:22] Because you don't have this massive migration process that, you know, is it was, is, was painful. ~~ ~~

[00:22:30] Michael Meyers: So that leads me to another question, you know, with the shift from, you know, 7 to 8, we went to semantic versioning, and we changed our numbering, which is confusing to people. I mean, a good thing, but all of a sudden you're seeing that numbers advanced quickly, you know, 8.30, 9, 9 is going to be 10 in a year and a half.

[00:22:48] Like, you know, the good news is that, you know, it really is just a numbering scheme. And the upgrade process is really seamless. It's nothing like what it was. You know, upgrade paths are, are super easy moving forward and that's phenomenal. That's, that's a huge reason to be adopting, you know, 8, 9 and 10, you know, the longevity you're gonna get out of it.

[00:23:11] But I've noticed, you know, like we're no longer supporting versions of 8. you know, like things are happening like, 8.whatever, no longer has security support. Like, can you give us a little insight into like how security support works within the 8, 9, 10 ecosystem?

[00:23:30] Michael Hess: So there's there's majors, minors, and patch releases then.

[00:22:38] So Drupal 7 is a major release. Drupal 8 is a major release. Drupal 9 is a major release. Drupal 10 will be a major release. The minor releases don't exist in seven. So let's take that off the table. Are the releases that come after that? So 8.1 , 8.x, 8.3, 8.4.

[00:23:56] The patch releases are 8.1.0, 8.1.2, 8.1.3. The, the, you know, there's a graphic that I can show you for this, that, that makes it a little easier to see what's happening. But basically we, you know, one of the benefits of the way we're running the release cycles is there is a, very, you know, fixed structure to releases, you know, with Drupal 7 and Drupal 8, you know, how many times was Drupal 7 delayed when it was initially coming out?

[00:24:28] How many betas were there? You know, there were not set releases and by the way, that's not, you know, I'm not criticizing. The way we did things, it was ready when it was ready. You know, what we're doing now is saying, okay, we have a very strict thing in place. If you want get it new feature in, it has to be at this stage before we're going to actually think about releasing it, this upcoming release.

[00:24:47] If it's not there, great, it gets pushed off to the next minor release. And so we've got this really or major release depending on what we're doing. and so, because the upgrade path is, is easy. The changes between 8.6 and 8.7 to do that release. Not that it's not difficult. And so, you know, the upgrading between minors is relatively smooth.

[00:25:13] You know, I'm not going to say it's been perfect. We've had one issue that I'm aware of and maybe more than one, but. Where, you know, there's been a headache involved, but you know, it's very clear what's in that minor release and what's not in that minor release, whereas in the Drupal 7, you know, the non semantic versioning routes, if you're doing, you know, I think we skipped like 10 points because we wanted to indicate there was a major change here.

[00:25:38] And so, you know, we, we just skipped random, like it was completely arbitrary. and so, you know the whole, and I'll share the link to what this looks like, so that you can share it with folks. You know, this really spells out exactly how it works when security coverage is how long security coverage is for each minor after it's released how the whole whole thing works.

[00:26:06] When we do betas it, you know, there's a, a fairly complex process in here, for doing this, but what's great about this is that it's predictable. Yup and so it's not a, you know, when the volunteer maintainer has the time to do this, there will be a release at these times. And if you want to know what's in it, there is a change log that will show you what's in the releases.

[00:26:32] There are release notes, not change log. I apologize - release notes that are detailed, that cover all the information that's in the releases. It is a, it is a much more predictable way of doing release management. But yes we do unsupport the previous minor six months after. the new minors release. And so, you know, effectively, if, you know, if I release 8.7 in May, people are still running 8.6, obviously 8.7 is now those supported, minor release.

[00:27:08] You get your support for 8.7 to do your upgrade through December. So you've got that time in there to go through and, you know, do those make that upgrade. And then of course, you know, you keep moving forward, but for the most part, you know, I've got a lot of Drupal 8 sites that I'm responsible for.

[00:27:26] I think, you know, I can count on one hand the number of issues with minor release updates, and you know, they're, they're, they're 99% of the time, very smooth.

[00:27:39] Michael Meyers: So the numbers are moving faster. The support windows are, are, are smaller, but the upgrade process is, is really easy. And it, you know, for so many reasons, security being paramount, you know, people should be keeping their sites up to date.

[00:27:57] And, you know, I just wanted to bring it up because I think, you know, it is, it's a lot to digest. It can be confusing. and it is, you know, in fact, a really good thing, a really easy thing. listen, Michael, you've been crazy generous.

[00:28:10] Michael Hess: I want to clarify with the security thing there, every minor release gets security coverage for a full year.

[00:28:19] Yep. that's typically six months of it being the active release branch. And then six months when the next minor is released, we, we keep that minor in support. The, the, you know, it's a predictable software that the other thing that's happening though, is you're getting new features.

[00:28:37] You can choose not to use them, but you are getting new features with these minor releases, which, you know, with Drupal 7, you know, you got new features like there, you know, it kept moving forward, but you know, you would get features, you'd get bug fixes, you get all sorts of things in one release, and it was hard to figure out what's going on there.

[00:28:57] This gives you a predictable time to test your release, to make sure you're happy with what you're going to release and that you're not breaking things in unpredictable ways.

[00:29:04] Michael Meyers: It's a much more mature model. I think, predictability planning, you know, it's, you know, sort of, yeah, it's, it's, it's good on the whole.

[00:29:14] So Michael, you've been, unbelievably generous with your time today. We went, we went way over. I, you know, there's so much more, I want to ask you. This is, I mean, I guess for a guy who's been involved in the community for a really long time, you gave me insight and answers to things that I've been wondering about for a really long time.

[00:29:29] So I hope that other people learned, as well, as much as I did. I'd love to have you back in the future because there's so many more things I wanted to cover. but this was, this was great. Really appreciate you joining us.

[00:29:42] Michael Hess: Thank you for having me. And I'm happy to come back and talk about the stories of how drupal.org almost used a commercial vendor whose name starts with an A for it to Git provider instead of GitLab. That's a fun story, but thank you so much. Have a great evening and a good weekend, even though it's a Friday. And I don't know what, when people are going to be watching this, but have a good evening.

[00:30:06] Michael Meyers: Thanks, Michael. And for the folks who tuned in, we're going to put all the links in the show notes here.

[00:30:12] Please remember to upvote, subscribe and share this out. You can check out our past talks at tag1.com/tagteamtalks. You can check out the end of life presentation. I mentioned the tag1.com/eol for end of life. As always, we'd love your feedback and suggestions. Let us know what you thought of the show, topic ideas for the future.

You can reach us at tag1teamtalks@tag1.com. Thank you guys so much for tuning in and joining us today. We'll see you soon. Take care.