Programming

Multi-file uploads in Drupal 8?

slashrsm — Sun, 03 Feb 2013 18:39:01 +0000

Multi-file uploads in Drupal 8? slashrsm Sun, 03.02.2013 - 19:39

EDIT (20.4.2013): This has been committed and is now part of D8 core.

Back at DrupalCon Munich code sprint I started to work on a core issue that is focused on implementing HTML5 multiple uploads support in Drupal. Since I've always been interested in file handling I immediatley found this very attractive. From the other hand I always strugeled when it comes to files in Drupal. There are so many solutions and none of them is really user friendly. I thought that this feature would make Drupal's UX much better. During my travel back home from Munich and the days after I spent some time working on the issue, but then it stuck somwhere in the middle due to my lack of time.

In January I started to work for Examiner.com where I got so-called Drupal day at the end of every sprint. That helped me to focus on this issue once again (and I am very thankful for that). During last 2 or 3 weeks I managed to finish the patch (in terms of features):

file and managed_file FAPI elements have been updated to accept #multiple option. Once this option is enabled element adds "multiple" argument to it's input tag, which enables multi-file uploads in recent versions of all major browsers. Old browsers should gracefully degrade to single-upload without any problems.
File & Image field widgets leverage new #multiple option in it's underlaying form elements to improve experience for multi-value fields. Users will no longer upload galleries on file at a time. Everything can be now done in a single step.
Existing tests were corrected to pass with newly added features (FAPI element behaviour changed a bit).

I've already been testing, but patch still needs a lot of testing and reviews. More information can be found in the issue queue.

Drupal's Plupload integration - file_validate()

slashrsm — Mon, 19 Nov 2012 08:39:36 +0000

Drupal's Plupload integration - file_validate() slashrsm Mon, 19.11.2012 - 09:39

Kevin Hankens posted an issue in Plupload's integration module a while ago. He complained about the way how module invokes file_validate() at uploads. After a bit of research I agreed with him.

Plupload module invokes file_validate() with file object that carries nonexistent file uri. Uri, that is passed to validation, is representing a location where file is supposed to be uploaded. At that moment the file is actually located on a completely different (temporary) location. Validation hooks are unable to load file and do various of operations as a result of this.

We propose a patch that fixes this. Problem is, that this solution also breaks API compatibility. As this issue definitely needs to be fixed, I'd still not like to commit something, that would make people willing to kill me. :)

I invite all developers with interest in Plupload to check this issue and provide feedback.

Relation Migrate integration and *reference upgrade path

slashrsm — Mon, 06 Aug 2012 21:35:19 +0000

Relation Migrate integration and *reference upgrade path slashrsm Mon, 06.08.2012 - 23:35

There was my patch committed to Relation module (which is great - if you have not tried it yet definitely do) today, that adds Migrate support to Relation and uses it to provide upgrade path from Term/User/Node/Entity reference fields to Relation. I hope that this feature will be useful to a lot of people. In order to use new functionality you'd need to enable Relation migrate, which comes together with Relation.

Migrate integration

Module implements class MigrateDestinationRelation, which is a destination plugin for Migrate. You can use this class in your custom migrate scripts in a very similar way as you'd use MigrateDestinationNode, which is most likely used when migrating nodes. The only difference will be in mapping, which should be done in prepare() or prepareRow(). This is required because of the complexity of relation_endpoint field, which requires a lot of information in order to properly build a relation. Here is an example of typical mapping definition:

public function prepare(stdClass $relation, stdClass $source_row) {
  $relation->endpoints[LANGUAGE_NONE] = array(
    array('entity_type' => $source_row->source_type, 'entity_id' => $source_row->source_id),
    array('entity_type' => $source_row->destination_type, 'entity_id' => $source_row->destination_id),
  );
}

This example is taken from actual implementation of *reference upgrade path in Relation migrate.

Term/User/Node/Entity reference upgrade path

A typical request by people thinking about using Relation on their existing sites is upgrade path from *reference modules. Migrate integration now allows us to do this. You shouldn't spend more than a few clicks to have this done. I will demonstrate this feature on a standard Drupal 7 installation. I have generated 50 nodes that are tagged with taxonomy terms. This is done via field_tags field, which is obviously a Taxonomy reference. I will first create new Relation type and configure it to allow relations from nodes to terms.

Now I have to configure Relation migrate. In order to do that I have to navigate to configuration page (found under Structure -> Relation types -> Migration). You'll find configuration for every type of reference field here. Currently we support Term (core), Node (References), User (References) and Entity reference. As I already mentioned I use standard installation for this demo, but other supported fields should appear here if there are any. Now I just have to enable upgrade path for field_tags and select relation type that should be used for this upgrade.

If I navigate to Content -> Migrate (Migrate UI must be enabled) i find four migration classes, each for every supported reference field type. I can see that I have some unmigrated term references, which are obviously tags on my nodes. I select this class and start the migiration process.

When migrate script completes it's job I end up with a lot of new relation, which are tags on my nodes.

Described functionality is currently available in 7.x-1.x-dev release of Relation module. It is relatively new and as such not very well tested. Please report back in the issue queue if you find any bugs or encounter any problems. I'd be glad to help. Hope you will enjoy Relation module and it's Migrate integration.

Strange behaviour (race condition) during node_save()

slashrsm — Tue, 10 Jul 2012 13:33:40 +0000

Strange behaviour (race condition) during node_save() slashrsm Tue, 10.07.2012 - 15:33

Problem

We have a bigger media website on Drupal 7 with an editorial team of ~20 members. Lately they started to complain about strange behavior when updating content on site. They said, that changes were not saved at all. If they opened a node form immediately after saving it Drupal served old (outdated) content. I first suspected Varnish, but it turned out that Varnish worked OK. Later I realized that the problem obviously lies in field cache.

Debugging

We use Memcached for most of cache bins on the site. When testing further I realized, that field cache returns outdated data even after the changes were saved. The strangest thing was, that everything worked well if I moved cache_field back to database. I first suspected bugs in Memcached itself or Memcache PHP extension. More testing convinced me that I was wrong. After A LOT of unsuccessful debugging I realized, that something sets field cache for a given node immediately after that node was updated (and field cache cleared). Data that was saved to field cache was unfortunately outdated, which resulted in a unsynchronized state of field cache. We had correct data in DB, but it's cached version was wrong.

At the end I realized, that issue happened because of a http call, that we implemented in one of hook_node_update() implementations. We have an outside service, that needs to know when we update our articles. To achieve this we called their callback, which caused their servers to immediately visit updated content. This generally happened while node_save() was still in progress. Diagram shows what was happening:

API docs for hook_node_update() states:

This hook is invoked from node_save() after the node is updated in the node table in the database, ...

That made me believe, that node was actually saved into DB, when my hook was called by Drupal. But that is not true.... If we look at the code of node_save(), we see that it uses db_transaction() (which is a smart thing to do):

function node_save($node) {
  $transaction = db_transaction();
  ....

But, what isn't so obvious is a fact, that every DB query called from node_save() or any function directly or indirectly called by it will not be executed immediately. It will be executed at the very end of node_save(), though. Docs for DatabaseTransaction confirm that:

This class acts as a wrapper for transactions. To begin a transaction, simply instantiate it. When the object goes out of scope and is destroyed it will automatically commit.

Taking this into consideration we have to correct our flow diagram:

And here was our problem.

Solution

One solution would be to do remote calls at cron. This was unfortunately not possible as our managers strictly requested that there should be no delay between content update and remote service call. Another solution was, to set some flag in form of a static variable and to call remote server in some other hook, that will happen later in a request flow. Not the best option, but it should work for now.

I also submitted a core issue about this, as I believe we should at least update docs for core hooks, to let people know about this fact.

The last mistery I had was about Memcached/DB. Why was this problem not present, when I used DB to store cache? The answer is quite obvious when you think a bit. Cache clear operation will be affected by database transaction, as you use DB to store cache. That means, that clear will be executed when all other queries are commited (at the very end of node_save()) resulting in outdated cache being deleted.

Have this happened to any other? How would/did you fix this at your site? I'd love to hear about other people experience....

JavaScript patterns

slashrsm — Wed, 02 May 2012 19:22:40 +0000

JavaScript patterns slashrsm Wed, 02.05.2012 - 21:22

https://shichuan.github.com/javascript-patterns/

slovenskenovice.si - biggest Slovenian Drupal site

slashrsm — Mon, 05 Dec 2011 22:15:27 +0000

slovenskenovice.si - biggest Slovenian Drupal site slashrsm Mon, 05.12.2011 - 23:15

Slovenske novice are Slovenian national daily newspaper with largest circulation on national level. Last week we lunched redesign of it's web variant, which is a news portal with about 200.000 unique visitors and 4.5 million impressions per month. This numbers wil definetely become larger soon, as they have monthly growth between 10 and 15%.

Site was developed on Drupal 7. We extensively used Panels and Views. Search is based on Apache Solr, driven by Search API framework. To achieve performance we used Memcached, Varnish, APC and developed some own cache expiration rules. This is how we achieved performance we needed and flexibility requested by editorial team. Everything is hosted on our own HA LAMP stack.

Just two days after launch we had national elections in Slovenia, which are also a big media event. Our site broke the record in daily visits and everything was working smooth.

Since a custom CMS, which was developed inside company about 10 years ago, was used before, one of the most important aspects of this project was experience for editorial team. They learned how to use Drupal surprisingly fast. There were some minor issues reported, but in general Drupal works really got for them.

Site was developed by core development team:

We also had some themeing help from Iztok Smolič (@Iztok).

We will publish some more information about project in the following weeks.

Survey about satisfaction with Scrum

slashrsm — Fri, 18 Nov 2011 22:30:00 +0000

Survey about satisfaction with Scrum slashrsm Fri, 18.11.2011 - 23:30

I am currently working on my bachelor thesis at Faculty of Computer and Information Science in Ljubljana. The topic of my thesis is implementation of Scrum methodology in a webdev department of the biggest national daily newspaper company, Delo.

I would like to compare my conclusions about it's strengths, weaknesses and challenges in implementing the Scrum methodology in real environment with the opinions of people who faced this method in their professional career. I therefore kindly ask you to participate in the survey, which will take you about 10 minutes.

I will share my conclusions here. Thank you for your time!

DrupalCon Denver 2011 - Media derivatives: take control over your files

slashrsm — Mon, 07 Nov 2011 21:45:07 +0000

DrupalCon Denver 2011 - Media derivatives: take control over your files slashrsm Mon, 07.11.2011 - 22:45

Derivatives API is a Drupal project I developed during this year's Summer of Code. It is a framework, that tries to implement a simple, reliable and powerfull framework for asset derivation in Drupal. It can cover a lot of different use cases. We are in the times of emerging new technologies of HTML5, mobile publishing, responsive web, ... I believe that Derivatives API has it's piece in a puzzle of support of this new technologies on Drupal.

In cooperation with my mentor, Kevin Reynen, I proposed a session at DrupalCon Denver, where we'd like to present this project to the wider public. Session will cover basic functionalities of the project, some possible use cases and it's extension with your own plugins.

Would you like to hear about this project on DrupalCon? Please vote for our session.

Do you have any suggestions or questions? Feel free to comment here or on session proposal.

Hope to see you in Denver!

How to add custom contextual links to views?

slashrsm — Tue, 20 Sep 2011 16:23:09 +0000

How to add custom contextual links to views? slashrsm Tue, 20.09.2011 - 18:23

Today I had a task to add nodequeue manipulation contextual links to some view. First I found this blog post, which explains, how to make desired menu items contextual links ready. As Marcus states in his post, I first needed to alter nodequeue menu items:

function MYMODULE_menu_alter(&$items) {
  $items['admin/structure/nodequeue/%nodequeue/edit']['context'] = MENU_CONTEXT_PAGE | MENU_CONTEXT_INLINE;
  $items['admin/structure/nodequeue/%nodequeue/view']['context'] = MENU_CONTEXT_PAGE | MENU_CONTEXT_INLINE;
}

Nodequeue admin pages were now ready to be added to any contextual links on site. Since Marcus explains how to alter contextual links on blocks, I had to figure out how to do it with views. Solution turned out to be very simple. Contextual links are added to view in views-view.tpl.php, where $title_suffix is displayed. I just needed to add nodequeue contextual links to that variable (which is a render array in fact).

function MYTHEME_preprocess_views_view(&$vars) { 
  $vars['title_suffix']['contextual_links']['#contextual_links']['sn_fpmain_news'] = array(
    // 3 is queue id in my case; it could be any other page argument 
    'admin/structure/nodequeue', array(3) 
  ); 
}

And here is the result:

Is there any better way to do this?

5 day challenge (i'm in!:)

slashrsm — Sat, 17 Sep 2011 21:09:01 +0000

5 day challenge (i'm in!:) slashrsm Sat, 17.09.2011 - 23:09

Today Jakub Suchy proposed a really cool thing on his blog. Idea is about spending 30 minutes each day in the next week to contribute something to Drupal. He proves, that only 63 people doing this, can do as much work as an individual could do in a month.

Since I immediately loved this idea, I decided to join. Here is my plan for #d5dChallenge:

Day 1: improve Media derivatives API views support,
Day 2: write some docs for Media derivatives API,
Day 3: work on YouTube uploader engine for Media derivatives API,
Day 4: spend some time in Media issue queue; find some issue to review or submit a simple patch; find some issues I could help with during Media code sprint,
Day 5: do some research about Rules integration development and plan Rules integration for Media derivatives API.

What do you think about this list? You can also propose something else, if you think I should rather work on some other tasks during this week.

Will you join too? I hope there will be even more than 63 of us!