Tag1 Consulting

Performance and Scalability Experts


yumrepos Puppet Module

Earlier this year we undertook a project to upgrade a client's infrastructure to all new servers including a migration from old Puppet scripts which were starting to show their age after many years of server and service changes. During this process, we created a new set of Puppet scripts using Hiera to separate configuration data from modules.

BDD: It's about value

I was drawn to Behavior Driven Development the moment I was pointed toward Behat not just for the automation but because it systematized and gave me a vocabulary for some things I already did pretty well. It let me teach some of those skills instead of just using them. At DrupalCon Amsterdam, Behat and Mink architect Konstantin Kudryashov gave a whole new dimension to that.

Tackling oversized cache items in Drupal

Drupal’s highly dynamic and modular nature means that many of the central core and contrib subsystems and modules need to maintain a large amount of meta-data.

Rebuilding the data every request would be very expensive, and usually when one part of the data is needed during part of the request, another part will be needed later during the same request. Since just about every request needs to check variable_get(), load entities with fields attached etc., the meta-data needs to be loaded too.

The pattern followed by most subsystems is to put the data into a single large cache item and statically cache it. The more modules you have, the larger these cache items become — since more modules mean more variables, hook_schema() and hook_theme() implementations, etc. And the same happens via configuration with field instances, content types and default views.

This affects many of the central core subsystems — without which it’s impossible to run a Drupal site — as well as some of the most popular contrib modules. The theme, schema, path alias, variables, field API and modules system all have similar issues in core. Views and CCK have similar issues in contrib.

With just a stock Drupal core install, none of this is too noticeable, but once you hit 100 or 200 installed modules, suddenly every request needs to fetch and unserialize() potentially dozens of megabytes of data. Some of the largest cache items like the theme registry can grow too large for MAX_ALLOWED_PACKET or the memcache default slab size. Since the items are statically cached, these caches can easily add 30MB or 40MB to PHP memory usage combined.

The full extent of this problem became apparent when I profiled WebWise Symantec Connect site (Drupal.org case study). Symantec Connect currently runs on Drupal 6, and as a complex site with a lot of social functionality has a reasonably large number of installed modules.

Memcached and PECL memcache on CentOS and Fedora

At Tag1 Consulting we do a lot of work on increasing web site performance, especially around Drupal sites. One of the common tools we use is memcached combined with the Drupal Memcache module. In Drupal, there are a number of different caches which are stored in the (typically MySQL) database by default. This is good for performance as it cuts down on potentially large/slow SQL queries and PHP execution needed to display content on a site.

Stop Disabling SELinux!

I see a lot of people coming by #centos and similar channels asking for help when they’re experiencing a problem with their Linux system. It amazes me how many people describe their problem, and then say something along the lines of, “and I disabled SELinux...”. Most of the time SELinux has nothing to do with the problem, and if SELinux is the cause of the problem, why would you throw out the extra security by disabling it completely rather than configuring it to work with your application?

Imported DISQUS Comments Not Showing Up On Drupal Nodes

DISQUS is a popular "social commenting" platform. It is integrated with many hosted blog platforms and open source CMSes, including Drupal. A client of ours exported the comments from their old Wordpress blog and then imported them into DISQUS. The problem was that the comments were showing up in the DISQUS dashboard, however, when you clicked their corresponding URLs, these imported comments did not appear in Drupal. While the Drupal module looks for comments on the node/X URLs, DISQUS was storing them at the old Wordpress URL which were implemented as path aliases in this case.

Tag1 Sponsors Narayan Newton For Drupal.org Infrastructure

Tag1 Consulting is sponsoring my work on Drupal.org Infrastructure. What this means is that instead of working on drupal.org whenever I can, I get to spend 20 paid hours per week on drupal.org infrastructure. In return for this, I have agreed to write a blog entry per month describing some of my work in detail. These will be entries covering security, performance, high-availability configuration and anything else interesting in my work on drupal.org. Hopefully these will be useful.

I look forward to spending more time securing and improving the performance of drupal.org and would like to thank everyone at Tag1 and our clients for this opportunity.

Tracking contrib and core patches with schema changes

During performance and scalability reviews of sites, we regularly find ourselves submitting patches to contrib modules and core to resolve performance issues.

Most large Drupal installations we work with use a variation of this workflow to track patches:

  • Upload the patch to an issue on Drupal.org if it's not already there
  • Add the patch to a /patches folder in revision control
  • Document what the patch does, the Drupal.org nid, and a reference to the ticket in the client's issue tracker in /patches/README.txt
  • Apply the patch to the code base

Tag1 At DrupalCon Chicago

We're excited to have 7 members of the Tag1 Consulting team attending the DrupalCon in Chicago next week. We are all looking forward to participating in another fantastic Drupal Conference. If you've not already bought your tickets, it's still not too late! Don't miss this one!

In Chicago, Tag1 will be passing out copies of Drupal Watchdog, participating in training courses, sessions, and BoFs, and generally enjoying the two-way sharing of knowledge with our fellow Drupal developers and users.

Drupal Watchdog Volume 1 Issue 1 Out The Door

Drupal Watchdog Volume 1 Issue 1 is on its way to Chicago to greet all attendees of the Chicago DrupalCon. Those that subscribed should receive the first issue within the next two weeks, possibly longer for international subscribers. This issue was made possible by 23 authors, 7 editors, 2 designers, and a 3-person layout team. Its 80 pages were printed on FSC Certified paper stock with soy inks.

 Chicago DrupalCon

I'll be in Chicago attending the DrupalCon, and eager to get feedback from other attendees. Questions I'm looking to get answered include: What did you most enjoy about Drupal Watchdog Issue 1? What do we most need to improve? Is there enough interest to publish quarterly in 2012? What digital formats are people most interested in? And, what topics would you be most interested in seeing covered in Issue #2?

The printer sent me a snapshot of some finished copies of Drupal Watchdog issue #1. The photo was taken from a phone and is not high quality, but it gives you an early sneak peak as to what you can expect:

 Drupal Watchdog

The copies of Drupal Watchdog that people find in their "tote bag" in Chicago contain an exclusive bonus: Examiner.com put together a full-color 28-page insert titled "#XMNR:CHI The Insider Guide To What To Do And Where To Do It". The insert suggests where to eat, where to drink, and what to see while you're in Chicago. It even includes a map of the area around the Drupal Towers and a list of food within easy walking distance.

Drupal Watchdog is a very content-rich magazine. Our first issue is 80 pages long and includes 7 feature-length articles, 13 standard-length articles, and 6 columns. Read on for the complete table of contents, it's quite likely you'll recognize a great number of our contributors! As a print magazine, Drupal Watchdog strives to bring you relevant information that remains useful to you for a long time to come.

Syndicate content