Dylan Clear

Tag1 Spotlight: Nedjo Rogers

Nedjo RogersNedjo Rogers is a Senior Performance Engineer with Tag1 based out of Victoria, Canada. He’s been an active Drupal contributor since 2003, has served as an advisory board member of the Drupal Association, and has led Drupal development projects for clients including Sony Music, the Smithsonian Institute, the Linux Foundation, and a number of nonprofit organizations. He’s also the co-founder of Chocolate Lily, where he builds web tools for nonprofits, including the Drupal distribution Open Outreach.

Nedjo and I chatted shortly after he returned from a bike trip, touching on a range of topics including his entry into Drupal, working at Tag1, his nonprofit distribution, Drupal 8, corporate influence, CMI, Backdrop, and mining activism.

Dylan: How did you get started in web development and Drupal?

Nedjo: In 1996 I was working at an environmental group focused on the impacts of mining. My first work was mapping--providing communities with information about proposed mining projects in their area and the impacts of existing mines. We also had an international project, working in solidarity with organizations and communities impacted by the activities of Canadian mining companies in Peru and other Latin American countries.

In 1997 I started building an organizational website using Microsoft Front Page. Then, working with an international network of mining activist organizations, I produced a complex relational database application to track the worldwide activities of mining companies. Our ambitious aim was to enable communities to pool their information, supporting each other. I built the database using the software I had installed on my work computer: Microsoft Access.

In short: I was trying to build liberatory, activist internet resources, but using proprietary tools. I didn't know better. I'd never heard of free software.

By 2003, when I was working at a small technical support NGO, I'd had my introduction to the free software community, thanks initially to Paul Ramsay of PostGIS renown. At the time, Drupal was a small and relatively little known project. The available contributed modules numbered in the low dozens. Compared to other options I evaluated, the core software itself did very little. But what it did do, it did well and cleanly. I set about writing and contributing modules and dabbling in Drupal core.

Dylan: Your Drupal bio says you have a masters degree in geography. When you started working in technology was it an explicit part of your job responsibilities or was it more of something you fell into because someone needed to do it?

Nedjo: Decidedly the latter. In my MA, I took exactly one course in anything related to technology: geographic information systems (GIS). I was teased initially when I started hacking code. "Nerdjo" was my nickname of that time.

I've always had a love-hate relationship with the idea of working in IT. In many ways, I love the work. At its best, it's a constant source of intellectual challenge. But technology per se I don't care about at all. Outside of the Drupal community, few of my friends know in any detailed sense what I do.

Dylan: Yet based on what I’ve read about you, I'd venture a guess that your education, your intellectual interests, your views have had a big influence on the work you’ve done in the Drupal community. Is that true?

Nedjo: Very much so. Software, code, documentation--my sense is that all of it begins with the basic questions we ask ourselves.

Like many others, I came to Drupal with a purpose: to build tools that empower local, change-making organizations and movements. To put emerging communication technologies into the hands of activists and community builders. To do so, we need to work with those communities to shape technology to the needs and abilities of those who will use it.

I believe of Drupal that it retains, at least latently, a lot of the original principles of its genesis as a student project: a core focus on communities and collaboration. So, for me, working to bring a grounded and critical perspective to topics like structure and architecture has been a core part of my service to the project.

Working at Tag1

Dylan: Can you tell me about how you started working with Tag1 and what your experience working here has been like?

Tag1 partner Jeremy Andrews and I were longtime colleagues at CivicSpace, and I worked closely with Tag1 manager Peta Hoyes when we were both at CivicActions, so when Peta reached out to me with the opportunity to work with Tag1 I already knew it was going to be a great place to work.

My first assignment was on archetypes.com, a complex custom build that included some intriguing challenges. The project came to Tag1 through a route that’s now become familiar. We were brought on to work on performance issues on the site, which was built originally by a different team. We recommended architectural improvements that would address performance bottlenecks and then were engaged to guide that process, working with the in house Archetypes dev team.

It was my first chance to work with Tag1 engineer Fabian Franz, and I was immediately struck by his creative genius. I also got to collaborate again with Károly Négyesi (ChX), who years earlier was an important mentor for me, encouraging me to participate in the planning sprint for fields in Drupal core. Károly was also on hand to witness my near arrest by a half dozen of Chicago’s finest on Larry Garfield’s mother’s back porch while attending a Star Trek meetup, but that’s a longer story ;)

Recent highlights at Tag1 include collaborating with staff at the Linux Foundation to upgrade and relaunch their identity management system and working with Mark Carver and others to complete a redesign for Tag1’s own Drupal Watchdog site.

Tag1 is a good fit with my interests and commitments and I love the chance to work as part of a creative team where I’m always learning.

Nonprofit Distribution

Dylan: You’ve developed a Drupal distribution for nonprofits, Open Outreach and a set of features called Debut. Tell me about those projects.

Nedjo: In 2010 or so, while working at CivicActions, I completed six months of intensive work as tech lead of a project for the Smithsonian, a site on human origins for the US National Museum of Natural History.

We'd repeatedly worked 12 to 16 hours days to complete a grueling series of sprints. The site was a resounding success, but for me it sparked a crisis of faith. If we were here for free software, what were we doing pouring our efforts into what was essentially closed, proprietary development?

More troubling for me personally was the realization that I'd steadily lost touch with the reasons that brought me to Drupal in the first place: the small, activist groups I care about and am part of. If I wasn't going to focus solidly on building technology specifically for them, exactly who was?

I left CivicActions and soon afterwards, my partner, Rosemary Mann, also left her job of ten years as the executive director of a small group working with young, often marginalized parents. With our two kids, we pulled up roots and spent five months in Nicaragua. While there, I volunteered with an activist popular health organization.

When we returned, both at a crossroads, we decided to start up a Drupal distribution for the activist and community groups that we cared about. Groups that, we felt, were increasingly being left behind as Drupal shops pursued high-paying client contracts.

When we started into the work, via a small partnership, Chocolate Lily, several people said they didn’t understand our business model. By focusing on small, low resourced organizations, we'd never generate the lucrative contracts that Drupal shops depend on.

And we said, yeah, that's kind of the point.

Our first Open Outreach site was for the organization that Rosemary worked for for ten years, the Young Parents Support Network.

When building Open Outreach, we put a lot of energy into interoperability. At that time, the Kit specification, aimed at enabling the features of different distributions to work together, was a recent initiative. We very intentionally divided our work into two parts. Everything that was general beyond our specific use case (small NGOs and activist groups) we put into a generic set of features, Debut. The idea was that these features could be used independently of our distribution.

We also put energy into various other efforts that seemed to be oriented towards interoperability. This included the Apps module, which, at least in theory, included interoperability in its mandate. Why put energy in this direction?

Features itself - focused on capturing configuration in a form that can be reused on multiple sites - can be seen as extending the "free software" ethos from generic platform tools (modules) to applied use cases. But so long as features were limited to a particular distribution, they were in essence crippled. They were proprietary, in the sense that they were tied to a given solution set. Only by explicitly tackling the problem of interoperability could we truly bring the powerful tools being developed in Drupal to organizations other than those that could pay tens or hundreds of thousands of dollars on custom development.

The original authors of Features at Development Seed recognized this problem and took steps to address it. But when they largely left Drupal development, their insights and efforts languished.

I won’t make any great claims for what we achieved with Debut. To say that it was not broadly adopted is definitely an understatement ;) But in my more optimistic moments I like to think of it as a step in a broader process that will take time to bear fruit.

As Victor Kane recently put it, "We truly need open source propagation of reusable sets of Drupal configuration, and to reclaim the power of generically abstracting solutions from single site projects with the aim of re-use, with the creation of install profiles and distributions."

Drupal 8

Dylan: You are getting at some issues that I want to circle back to shortly. I’d like to shift to Drupal 8. What do you find most exciting or interesting about Drupal 8?

Nedjo: I've written or drafted several contributed modules now with Drupal 8. Coming from previous Drupal versions, I found it challenging at first to wrap my head around the new approaches. But as I got comfortable with them, I gained a lot of appreciation for the basic architecture and the flexibility and power that it enables.

Only a couple of months in, I already feel like I'm writing my cleanest Drupal code ever, thanks to the solid framework. While the previous procedural code left a lot up to the individual developer in terms of how best to structure a project, Drupal 8's well developed design patterns provide a lot of guidance.

One key feature of Drupal 8's architecture is "dependency injection", which roughly means that the elements (classes) that a particular solution needs are passed in dynamically as customizable “services” rather than being hard-coded. This design choice means a considerable leap in abstraction. At the same time, it opens the way for very specific and efficient customizations and overrides.

A concrete example, returning to the question of interoperability. A key barrier to features being interoperable is conflicting dependencies. This issue arises from the fact that some types of components need to be supplied once and used in many different features. Examples are user roles and "field bases" in Drupal 7. (In Drupal 8, what was a field base is now a "field storage".) In practice, most distributions put these shared components into a common feature that becomes a dependency for most or all other features in the distribution. For example, two different distributions might both include a core feature that provides a “body” field and an “administrator” role and that is a dependency of other features in the respective distributions.

This means you can’t easily mix features from one distribution with those of another, because their dependencies conflict.

In Drupal 7, I wrote the Apps Compatible module to try to address this problem. Apps Compatible makes it possible for multiple features and distros to share a certain limited set of shared configuration without conflicting dependencies. However, due to architectural shortcomings in Drupal 7, it does so via a bunch of awkward workarounds, and only for a very limited set of types of components (entities).

In Drupal 8, I've drafted an equivalent, Configuration Share. Using the dependency injection system, I was able very lightly to override a single method of the 'config.installer' service. In doing so, I achieved the same aim I’d attempted in Apps Compatible--except that this time it was done cleanly, in a few short lines of code, and for all types of configuration types at once.


I also wrote the initial architecture of the Drupal 8 Features module, which - thanks to some fresh approaches building on Drupal 8 architectural improvements - promises to be a vast improvement over previous versions. Mike Potter at Phase2 is taking my start and adding on a lot of great ideas and functionality, while respecting and extending the basic architecture.

So, yeah, so far I’ve loved working in Drupal 8. I honour the creative energy that's gone into it. At the same time, I have a critical perspective on some directions taken in the version and the project as a whole. To me, those perspectives are not in any way contradictory. In fact, they stem from the same set of values and analysis.

Corporate Influence

Dylan: That’s a good segue back to an earlier point you made that is worth revisiting. A not-insignificant portion of early adoption and funding of Drupal work was driven by small, scrappy nonprofits, working with dev shops with social change in their missions. Many of those early adopting orgs were attracted to Drupal not only for the economic accessibility of an open source CMS but for the philosophical values of community and collaboration and mutual empowerment through shared interest. You’ve been vocal about your concerns regarding enterprise influence. Assuming that Drupal does not want to lose these users, what can or should be done?

Nedjo: You're correct to highlight the important and ongoing contributions of contributors and groups motivated by the ideals and principles of free software and more broadly of progressive social activism. We can think here of CivicSpace, a social change project that was a key incubator for a lot of what shaped Drupal core.

Back in 2009, working at CivicActions, I had the rare opportunity of doing six months of solid work on Drupal core and contrib for a corporate sponsor, Sony Music.

As technical lead of a fantastic team including Nathaniel Catchpole and Stella Powers, and in close collaboration with great folks at Sony Music and Lullabot, I got to dig into practically every area of Drupal code related to language support and internationalization. At the end of the project, on my own time, I wrote a detailed case study crediting Sony Music with their amazing contribution.

So, when it comes to corporate sponsorship, you could say I'm an early adopter and promoter.

Just not an uncritical one.

In a thoughtful 2008 post, Acquia co-founder Jay Batson wondered whether the balance between volunteer and commercially sponsored development “will change for Drupal as Acquia appears.” His starting point was a blog post on “The effects of commercialization on open source communities”.

Jay bet that sponsored development would not become “the most”, while, commenting on Jay’s post, Drupal founder and Acquia co-founder Dries Buytaert was “100% confident that the Drupal community will continue to do the bulk of the work.”

Dries noted though that “What is important to company A, is not necessarily important to company B. … At Acquia, we have our own itch to scratch, and we'll contribute a lot of code to scratch it properly ;-)”

I recently read a relevant article from 2011, “Control in Open Source Software Development”, in an industry journal. The authors outlined a series of strategies that companies could follow to ensure they achieved significant control over open source projects and thus "help companies achieve their for-profit objectives using open source software."

Looking at the strategies examined, we can see many of them at work in the Drupal project, including sponsoring the work of lead developers.

The question is not whether the “itch scratching” Dries described affects the functionality and focus of the project. Of course it does. That’s the whole point.

It’s how.

Besides sponsorship, there are also indirect but potentially equally powerful influences. Through our work, we're exposed to certain types of problems. If, as our skill level and reputation rise, we end up working on larger budget projects for large corporations, governments, and enterprise-level NGOs, we develop expertise and solutions that address those clients' priorities. This focus can carry over in both explicit and more subtle ways to our work in other areas, even if we're doing that work as volunteers.

I see this influence on a daily basis in my own work. I couldn’t count the number of customizations I’ve done for paid client projects and then carried forward as a volunteer, putting in the unpaid time to turn them into a contributed patch or module.

So, seven years on, it’s worth returning to the questions Jay raised. Have we as contributors in effect been relegated to a role of outsourced and unpaid bugfixers, or do we retain a central role in shaping our collective project? What is our project? What do we want to achieve with it--ultimately, who it is for?


Dylan: You’ve raised some concerns about the direction that the Configuration Management Initiative (CMI) in D8 is going...

Nedjo: Yes, I think we can see a lot of these questions - of influence, target users, and enterprise interests - at play in Drupal 8 configuration management approaches.

When he introduced Drupal 8 initiatives in his 2011 keynote address at DrupalCon Chicago, Dries explained how he’d consulted twenty of the largest companies and organizations using Drupal about their pain points, and configuration management was one of two things that stood out. So from the start, the focus of the CMI was, as Dries put it, to fix “the pains of large organizations adopting Drupal”.

The CMI began with a particular limited use case: the staging or deployment problem. Initiative lead Greg Dunlop was explicit about this focus. Greg hoped that further use cases - like those of distributions - could and would be added as we moved forward.

Recently, I began tackling in a serious way the possibilities and limitations of Drupal 8 configuration management when it comes to distributions. My starting place is a very practical one: I co-maintain a distribution, Open Outreach, that's aimed at small, low-resourced organizations. I need to know: will Drupal 8 be a sound basis for a new version of our distribution?

As I dug into the code, two things happened. First, as I mentioned before, I was impressed with the architecture of Drupal 8. Second, I was concerned about implications for small sites generally and for distributions in particular.

Briefly (and please bear with me, this gets a bit technical): In Drupal 7, the system of record for exportable configuration like views or image styles provided by a module is, by default, the module itself. If a site builder has explicitly overridden an item, the item’s system of record becomes the site (in practice, the site database).

An item that has not been overridden is on an update path and automatically receives improvements as they are added to the original module. An overridden item is in effect forked--it no longer receives updates and is maintained on a custom basis. This model prioritizes the free software sharing of updatable configuration items among multiple sites, while enabling customization via explicit overriding (forking).

In Drupal 8 as currently built, the system of record for all configuration is the site. There is no support for receiving configuration updates from modules. In effect, the free software use case of sharing configuration among many sites has, for the moment, lost out to the use case that's built into Drupal core: staging configuration between different versions of the same site.

While staging can, in theory, benefit anyone, in practice, there are pretty steep technical hurdles. To fully use the staging system as currently built into Drupal 8 core, you need to have access to a server that supports multiple databases and multiple install directories; know how to set up and manage multiple versions of a single site; move configuration and modules and files between different versions of the site; and make sense of a UI that presents machine names of configuration items and, even more dauntingly, line-by-line diffs of raw data representations of arbitrary types of configuration.

In other words, the configuration management system - which includes a lot of great ideas and improvements - serves a staging use case that has a lot of built-in assumptions about the technical capacity of the target users, and so far lacks support for the more broadly accessible use case of receiving updates to configuration supplied by extensions (modules and themes).

It looks to me like a case where focusing too closely on a particular use case (staging) has raised some unanticipated issues for other use cases, including distributions.

None of this is in any way an irreversible problem. The use case problem is one that Greg Dunlap pretty much foresaw when he commented on his initial limited focus. He noted it was only a starting place, with lots of followup to do. The contributed Configuration Update Manager module provides a good start. I've sketched in potential next steps in Configuration Synchronizer.

But my view is all of this belongs in core. Only there can we ensure that the free software use case of sharing configuration among multiple sites is on at least an equal footing with that of staging config on a single site.


Dylan: Is Backdrop part of the answer?

Nedjo: It's a welcome addition and a healthy sign. For Open Outreach, we're evaluating Backdrop alongside Drupal. We're even considering building versions for both.

Backdrop inherits some of the same issues as Drupal 8 when it comes to configuration management. But there are encouraging signs that the lead Backdrop developers are both aware of these issues and open to fixing them in core.

Mining Activism

Dylan: You mentioned that mining activism was your entry point into web technologies. Are you still involved in activist work?

Nedjo: I was out the loop for several years in terms of mining activism, but recently I've gotten involved again. Three years ago I co-founded a small organization in my hometown of Victoria, the Mining Justice Action Committee (MJAC). Last month I coordinated a visit by a Guatemalan indigenous leader, Angelica Choc, whose husband was killed in connection with his resistance to a Canadian mining company's operations. Angelica is part of a groundbreaking legal case to hold the company HudBay Minerals accountable for impacts related to its presence in Guatemala. She was a focus of the film Defensora.

At the event, I provided interpretation from Spanish to English as Angelica shared her story with the more than 100 people who filled a church hall. As I spoke her words, I was in tears. From the pain that she and family and community have suffered, but just as much from her courage, vision, and resilience.

My partner, Rosemary, is also an active member of MJAC, and last year she built its organizational website using our distribution, Open Outreach. In small ways, circles are completed.

Dylan: Last question. How was your bike ride?

Nedjo: My younger child is eighteen now and plans to leave home this summer, so these last months with them at home have a special intensity. Ardeo and I biked along a former rail line dotted with trestles that carried us high above the Cowichan River, swollen with spring runoff. In several places, winter storms had brought down trees that blocked the trail and we had to unhitch our panniers and work together to hoist bikes over the sprawling trunks. Dusk was falling as we reached our campground beside the river.

These times of connection - personal and ecological - are so essential. They remind us that technology, or our work in the world, is at most a detail, a small means to help us realize what really matters. What truly has value.

You seem to have caught me in a philosophical mood. We'd better stop here before I sound more sappy than I already do!

Dylan: Thanks for your time, Nedjo!