Saturday, April 12, 2014

"General Type": a new criterion for Plone Collections

A new 1.2 version of plone.app.querystring has been released.
There are some improvement and bugfix but I'm particularly interested in one new feature: customizable parsed query. Why?

Some times ago I started developing a new product for providing some usability enhancement in type categorization using Collection but it was a dead end: it wont work without patching Plone. But the accepted pull request changed everything, so here it is: collective.typecriterion.

The product want to replace the Collection's "Types" search term (adding a new search term called "General type").

The scope of the add-on is to fix some usability issues with Collections:
  • Users don't always understand all of the content types installed in the site
  • User don't always get the difference from a type to another (classical examples: Page and File, or File and Image)
Plone type collection criteria Also there are some missing features:
  • There's not way to quickly define a new type alias or exclude types from the list
  • There's no way to group types under a general (but more user friendly) new type
Some of the point above could be reached searching types using interfaces (through object_provides index) instead of portal_type (the attribute that commonly store primitive type name of every content, but:
  • although search by interface is the suggested way to search by types, it's not used anywhere by Plone UI
  • using interfaces lead to inheritance behavior (which is great... until you really want it)
  • sometimes you don't have the right interface to use. For example, there's an ITextContent interface in ATContentTypes, but it's implemented only by Page and News, not by Event. And generating new interfaces is a developer task
The idea is to keep using portal_type but give administrators a way to group and organize them in a more friendly form.

After installation the new control panel entry "Type criterion settings" will be available.
Plone general type control panelThe target of the configuration panel is simple: is possible to group a set of types under the cloak of a new descriptive type name. In the example given in the image we take again definition of a "textual" content (a content that contains rich text data), grouping all the know types.

After the configuration you can start using the new search term.
Plone type collection criteria Usability apart there's also another advantage in this approach, that is the integration with 3rd party products.

Let say that you defined a new general type called "Multimedia" and you configure it as a set that contains Image and Video, and let say that Video went from the installation of redturtle.video product.
After a while you plan a switch from redturtle.video to wildcard.media. What you need to do is simply to change the configuration of the general type, not all the collections in the site.

Finally an interesting note: the code inside the collective.typecriterion is really small. All the magic (one time again) came from Plone.

Thursday, February 20, 2014

Plone, HTML 5 Canvas and Face Detection with Webcam

One of my latests articles was about HTML 5 Canvas and Webcam integration and in the same article I put all together in a Plone add-on for integrating portrait changes with Webcam.

Recently, following a retweet of one of the cool guys I follow on Twitter, I randomly hit an article that talk about a game that integrate user's webcam as a controller (unluckily I lost the original link, neither I remember the programming language used. The article were also generally introducing face recognition, and this captured my attention.
I asked myself how cool could be getting a face recognition feature with Python. It this something possible to do? Probably not so easily...

Introducing OpenCV

...or not? If you look for "face recognition" and "Python" on Google you'll always get references to OpenCV, the Open Source Computer Vision Python library.
I simply scratched the surface of this huge piece of technology as I'm totally a newbie about computer vision. What I get is that the library can really do a lot of stuff, and it's well documented.

Let me introduce some very-general information.

To use OpenCV for detecting faces on an image you must understand the difference between "face recognition" (find a know face on image or video) and "face detection" (find a face, in general). Obviously the second is a simpler task. Why? Because whatever is your task, OpenCV must be trained to find your target. For not simple task like face recognition you can (for example) train the software by providing a set of images where the object face can be found, and the result of the train is an XML file. After that you can you this file to implement something like Picasa, iPhoto or Facebook are doing when you submit new photos.

With face detection things are simpler because you can find one of those XML file online, already generated for you.

Another important informations about the library: recently a deep API changes has been performed so a lot of examples you can find online are broken (or must be fixed).

Finally, when you are able to do so, prefer the use of cv2 library instead of cv. They are more or less the same library but (for what I understand) cv2 is faster because is based on numpy, so C-compiled code.

Going back to what i did: I focused on face detection. Let see how.

Applying face detection to Plone (yes, I said it)

Meanwhile I were also fixing some minor issues in collective.takeaportrait. A new minor feature is the possibility to move the viewfinder by using mouse drag&drop (because must be forced to be in the middle of the screen is not comfortable).
Here came The Idea: how about a viewfinder that automatically center onto the face captured by the webcam?

Here my wish list:
  • JavaScript check for server side availability of face detection feature (just because OpenCV is not a simple library to be installed... and let me be honest: this is a cool feature, but not really useful)
  • With a not-so-long delay the whole Webcam image take by the canvas is sent to the server for a face detection view
  • OpenCV on the server perform the face detection
  • If a face is found, a rect is sent back to the JavaScript callback
  • The viewfinder is centered on the face using the feature already implemented for drag&drop
The use of Plone here it's a bit unnatural but It was the simplest environment for my experiment because of the work already done with the webcam in the last article. Apart Plone itself I hope you'll the general idea: how simple can be the browser/webcam/face-detection integration, whatever will be your back-end Python framework.

As you can suppose, the experiment was a success!

Introducing collective.takeaportrait face detection feature

I don't think I can add more useful details. Let's see the video!

Sunday, November 24, 2013

No more pdb.set_trace() committed: git pre-commit hooks

After 1, 2, ... 5 times it happens, you must find a way to solve the problem.
I'm talking of committing to your git repository a pdb.set_trace() you forgot to remove.

What is really nice of git (something missing in SVN for what I know) is the support to two type of hooks: client side and server side.
While server side hooks are complex and triggered when you push to the repository, client side hooks are simpler and under your control.

So, I can use client side git hook to solve the pdb problem?

Git hooks are executable file (of any kind) inside the .git/hooks directory of your repository. Those files must have a precise name, that match the action they capture (you can find a complete list in the Git Hooks reference).
In our case: we need to use an executable named pre-commit.

As you probably noticed, hooks are inside the repository. But can I have a centralized database of hooks that will be replied in every repository?
Yes, you must use Git templates.
To quick set you global templates, see this nice article: "Create a global git commit hook".
For impatiens, in two lines of code:

$ git config --global init.templatedir '~/.git-templates'
$ mkdir -p ~/.git-templates/hooks

Now put the executable file named pre-commit inside this new directory. After that, when you create a new repository, hooks inside the template directory will be replicated inside the repository.

The pdb commit hook is a well know problem, already solved. The best reference I found is in the article "Tips for using a git pre-commit hook", that use a simple bash script.

I simple changed the author idea a little, because I don't want to block commit when a commented pdb is found in the code (bugged! See next section!):

FILES_PATTERN='\.py(\..+)?$'
FORBIDDEN_PATTERN='^[^#]*pdb.set_trace()'
git diff --cached --name-only | \
    grep -E $FILES_PATTERN | \
    GREP_COLOR='4;5;37;41' xargs grep --color --with-filename -n \
    -e $FORBIDDEN_PATTERN && echo 'COMMIT REJECTED Found "pdb.set_trace()" references. Please remove them before commiting' && exit 1

There are some other info you'd like to know about client side hooks:
  • You can ignore the hook for a single commit (git commit --no-verify)
  • Only a single hook can exits in a repository. This is a limit, however you can find workarounds.

EDIT (21 December)

Normally I don't modify old articles, but I found a bug in the solution above. The problem: grep command return non-zero status code when it found no match.

Here another working solution (I'm not a Linux bash ninja... any other cleaner suggestion will be accepted!):

FILES_PATTERN='\.py(\..+)?$'
FORBIDDEN_PATTERN='^[^#]*pdb.set_trace()'
git diff --cached --name-only | \
    grep -E $FILES_PATTERN | \
    GREP_COLOR='4;5;37;41' xargs grep --color --with-filename -n \
    -e $FORBIDDEN_PATTERN && echo 'COMMIT REJECTED Found "pdb.set_trace()" references. Please remove them before commiting'
RETVAL=$?
[ $RETVAL -eq 1 ] && exit 0

Saturday, November 16, 2013

Dive into HTML5 Canvas

In last weeks I read a book about the new canvas element of HTML 5: HTML 5 Canvas, by Steve and Jeff Fulton
I don't want to review the book itself (just two word: is ok) but reading it lead me think about "how Canvas can change the Web in future".

First of all: I found all the HTML 5 Canvas feature interesting, but while reading the book I felt a sort of deja-vu. Every time I run one of the given example about drawing and painting, it was like I had already see it on a Web browser...
No, I'm not talking of Flash (I never had any experience with that) but about Java Applet! Yes, I really said "Java Applet"!
When I started Web development 10 years ago applet was "cool" because you were able to do a lot of stuff impossible to to with pure HTML + JavaScript. But you already know that applet failed.
Now using canvas you can do (again) a lot of drawing work, but still: how can this be useful to Web users?

HTML 5 Canvas: The Good Part

I was not able to find every answers myself, so I asked to Twitter and I get back some useful feedback..
  • Videogames development. A lot of this. In my opinion this is great! I don't have time for developing videogames anymore, but is still a discipline I like to follow. A lot of new JavaScript frameworks for game development are coming out, thanks to canvas.
  • Graphs and Histograms. We have a lot of powerful graph generator for server side languages, that can create images on the fly, but we can now stop query a remote server for that (remember that HTML 5 means also "bein offline").
  • File handling and preview. As HTML 5 is able to manage files, and canvas can also manage some type of media like images, audio and video (see below), plugins like jQuery File Upload are now possible.
I get some more good examples, but in the meantime I continued reading the book, and I found new super-cool answers myself! So: keep reading.

HTML 5 Canvas: The Super Fancy Cool Part

First of all: video!
I don't want to simply talk about HTML 5 video support or how you can control video with JavaScript, but what you can do with video and canvas.
This can be sum in a simple sentence: canvas can handle a video like it were an image. What does it means? You can draw the current video frame (taken from a standard HTML video element) inside a canvas; write the current video frame every 20 milliseconds and you are really playing the video inside it.

Why is this cool? I can already see a video element inside my HTML... so?
The great part is that you can draw a video frame inside the canvas then draw other stuff on the top of it! You can add comments and images on the running video! Wow!

And now the best part: HTML Media Capture API.
A lot of browser already support this new technology that make possible from JavaScript to access native webcam (and microphone) of the user's device. And how JavaScript can use this privileged access is not putting it into canvas?
And after that? Can I upload my work to a server? Yes!

Playing with this new toy I spent some times developing a new Plone add-on: collective.takeaportrait. This sum all the stuff I learned about media capture and video manipulation:
  • If the browser support getUserMedia call, a new button appear
  • The button will open an overlay where the webcam output is displayed
  • A viewfinder with the standard Plone ratio for user's portrait and a countdown are drawn over the streaming video
  • Use can save a photo and send it to the server for replacing the current portrait

This is something that you already see somewhere, some social networks (for what I remember, Facebook for sure) already give that chance to users, but it's really amazing to see how few lines of JavaScript code can raise the usability of your site!

Now the bad news (also reported in the Fulton & Fulton book): there's no support for those features inside mobile devices right now. Really sad.

Saturday, October 19, 2013

Reusable jQuery plugins with Bower

Inspired by the new article of Maurizio Lupo named "Reusable javascript modules with Bower" and by a recent discussion we had at work about modern front-end development (mainly focused on the Plone world), today I took some minuted to test bower.

To quickly explain bower to a Python developer I can say that it's "the Distutils of the JavaScript world".
I'm not a front-end developer because I think that "knowing how to do some JavaScript" doesn't mean being a front-end guy, however I like the direction where JavaScript is going. But note: take the rest of this article like it is: a "Bower for dummies" note!

In my last article I quickly introduced the jQuery Plugin site way of keeping update it's database: a simple JSON file. Bower is doing the same for populating its component registry site.
So an "official" jQuery plugin can be also a bower component.

Step by step

First of all you need to install the node package manager (npn), and for a MacOS user is really simple (same for Linux guys):

    $ sudo port install npm

After that you can install bower.

    $ npm install -g bower

No we'll go back to our jQuery plugin.
First of all you need the bower.json file, but instead of writing it manually, let's simply type...

    $ bower init

... and answer to the questions. After that you can go inside the file and add some other missing stuff by looking and the available syntax.

This is a possible result:
{
  "name": "waria-checkbox",
  "version": "0.2.2",
  "homepage": "http://plugins.jquery.com/waria-checkbox/",
  "description": "jQuery WAI ARIA Compatible Checkbox Plugin",
  "main": "jquery.waria-checkbox.js",
  "keywords": [
    "wai-aria",
    "accessibility",
    "js"
  ],
  "authors": [
    "Luca Fabbri <luca@keul.it>"
  ],
  "license": "MIT",
  "ignore": [
    "**/.*",
    "node_modules",
    "bower_components",
    "test",
    "tests",
    "CHANGES.rst",
    "README.rst",
    "demo.html",
    "LICENCE.rst",
    "README.rst",
    "waria-checkbox.jquery.json"
  ],
  "dependencies": {
    "jquery": ">=1.7"
  }
}

The bower.json file is really similar to the file needed by the jQuery plugin site, but it's not the same (a boring task: you must keep updated both files).
As we are focused on jQuery plugin, note that jQuery (that can be installed using bower) is defined in the "dependencies" section.

Finally you need to create a new git tag:

    $ git commit -am "Now is possible to install the plugin by using bower"
    $ git tag -a 0.2.2 -m "Tagging version 0.2.2"
    $ git push --all
    $ git push --tags

Last step: register the plugin onto the bower registry:

$ bower register waria-checkbox https://github.com/keul/jquery-waria-checkbox.git

Installation

    $ bower install waria-checkbox
    bower cached         git://github.com/keul/jquery-waria-checkbox.git#0.2.2
    bower validate       0.2.2 against git://github.com/keul/jquery-waria-checkbox.git
    bower cached         git://github.com/components/jquery.git#2.0.3
    bower validate       2.0.3 against git://github.com/components/jquery.git#>=1.7
    bower install        waria-checkbox#0.2.2
    bower install        jquery#2.0.3
    
    waria-checkbox#0.2.2 bower_components/waria-checkbox
    └── jquery#2.0.3
    
    jquery#2.0.3 bower_components/jquery

Now inside the bower_components folder we have both the plugin and jQuery.

Sunday, September 1, 2013

Extending jQuery selectors and facing conflicts with querySelectorAll

You know that is possible to extend the jQuery selector capabilities?
Just type...

$.expr[':'].foo = function(element) {
    ...
};
... and this function will be called when :foo selector is used.
Nothing new on this side.

Recently I started working on a new jQuery plugin and to make things simpler I needed to find a way to override an existing jQuery selector. Again: nothing new; some years ago I found that the method described in the article above can be also use for override, and not only for extend.
To make things more testable I decided to move this behavior into another (separated) jQuery plugin, which is the argument of this post.

When I used this method again nowadays I found unexpected results.
I was looking a way to change the way :checked and :checkbox selectors work, so I defined...

$.expr[':'].checkbox = function(element) {
    ...
};
...and...
$.expr[':'].checked = function(element) {
    ...
}; 
This is what I found:
  • :checkbox was working as expected
  • :checked was not working as expected
In facts, the :checked selector was only working when using it inside a .filter() call.
I'm not a jQuery core expert, I never looked at it's code very much, but this time I needed to investigate my problem. Also: I need to make this work on jQuery 1.7 and more modern 1.10 version, and codes are quite different.

Here what I found: both jQuery versions contains a method for capturing the :checked selector, but it's only called when you call filter() (so here my override attempt works as expected). For normal selectors jQuery now heavily relies on native querySelectorAll API for every browser that is supporting it.

This is the core of the problem: the :checkbox selector is a non-standard ones (not defined by any CSS specifications) while :checked is a know CSS selector. So browsers that support the :checked selector for querySelectorAll are calling this native API.

This is someway hilarious! While querySelectorAll are making our browser (and jQuery usage) faster, it's lowering jQuery extensions capabilities.

I found no smart way to change how querySelectorAll works (and probably there's no way at all, I think we are at C compiled code level here).
The trick I used is to disable the native querySelectorAll when the selector contains :checked, and in that case call my jQuery version instead.
How?

    var pattern = /\:checked/;
    if (document.querySelectorAll !== 'undefined') {
        // need to partially disable querySelectorAll for :checked
        document.nativeQuerySelectorAll = document.querySelectorAll;
        document.querySelectorAll = function(selector) {
            if (pattern.test(selector)) {
                throw('Native ":checked" selector disabled')
            }
            return this.nativeQuerySelectorAll(selector);
        }
    }

The first step is to disable (keeping a "backup") querySelectorAll, changing it with a custom function. Then all I need to do is check if the :checked selector is used somewhere in the query. If not: just call the backed up querySelectorAll, but if it's called somewhere I simply need to raise an exception.

This is the interesting part I found looking at the jQuery source: jQuery core try to use querySelectorAll every time is possible, switching to internal JavaScript code only when it's not supported. In this way I'm simulating the fact that my modern browser is not supporting :checked selector for querySelectorAll.

Problems

I found this experiment interesting, but there's some things I don't like:
  • I'm disabling the native querySelectorAll usage also when it's basic features are enough (i.e: if I really need to load only checked checkboxes and not other fancy stuff)
  • I'm wrapping every querySelectorAll calls inside the selector check, making all calls to querySelectorAll slower
  • I want only to extend jQuery here, but I'm also disabling all native querySelectorAll calls if the query containes ":checked"
  • I'm changing how JavaScript works. This is calling me back to times when I used prototype.js (that I never liked)
Any suggestion for a better code way are welcome!
This is the result of the experiment: jQuery WAI ARIA Compatible Checkbox Plugin.

Off-topic

Apart the problem described in this article, I let's spend some words on the "new" jQuery Plugin Site. Last time I published a jQuery plugin (lot of time ago) this site was a total mess: you need to authenticate, upload source tarball, write documentation on a wiki-like page, ...

I really liked how it works now: if you want to publish a plugin, just put it on GitHub and configure a commit hook! Simple and amazing!

Wednesday, May 1, 2013

How to make your Plone add-on products uninstall cleanly

This article contains practical informations, but also some personal opinions, about Plone install and uninstall tools. You are warned!

Preamble

Recently I worked on lot of Plone migration from Plone 3.x to versions 4.2.

Is Plone migration to newer version a difficult task? Not on its own. The Plone CMS is distributed with an internal migration procedure that simply works! What can make the difference are add-ons products, that sometimes transform this one-click operation something difficult.

When you have migrate a Plone site you'll probably find some annoying issues:
  • add-ons that must be removed because never used by customer
  • add-ons that must be removed because newer products are giving the same features
  • add-ons that must be removed because they are totally unmaintained now, and you must migrate to something else
Whatever is your motivation, sometimes Plone product removal can be a complex task and not for a Plone internal technological problem but because of programmers laziness.

Once Upon a Time: before Generic Setup and ZCA

When I started working with Plone making a Plone product "installable" was a Python task: add-ons contained an Extension folder with an install.py (or Install.py, uppercase) file and an install function inside.
This function did all the magic stuff (registering types, new properties, adding configuration stuff, ...).
The problem with that approach was that every Plone programmer was using a "personal style" for doing same things: obviously low-level APIs used were the same so, more or less, different product's install procedures were similar, but not exactly the same.

If you are curios, for historical reason take a look at an example: the oldest version of Poi install code.

Generic setup

Generic Setup was introduced with Plone 2.5. For the first time a (initially limited) set of common operation were translated to an XML format: for example you are now able to describe the new-type configuration using an XML file instead of writing Python.

Another example: a newer (still very old) version of Poi code.

Nowadays the main problem of Generic Setup is the lack of a central documentation (however the Plone Developer Manual is going in the right direction), but looking at the Plone source you can learn a lot.
Personally, I always look at the Products.CMFPlone source when I don't know/remember something but this can't help you for learning uninstallation.

ZCA

Meanwhile Plone 2.5 introduced also new Zope 3 features (everyone were talking of Five).
This gave us a lot of really cool things we have today (browser views, viewlets, events, utility, ...)... and new "persistent problems" to face (portlets, persistent utilities, marker interfaces, ...).

This was an important moment for Plone features, but at the same time the always-present ZMI started losing it's superpowers. What I mean: before Plone 2.5/Plone 3 the Zope Management Interface was a very powerful tool where you was able to fix things and clean garbage; nowadays ZMI contains more or less the same features of years ago.

Please note that using ZMI had never been a good way of fixing things or configuring your Plone site: a good Plone add-ons must configure itself when you install it, and fully remove itself on uninstall.
But usage of ZMI helped armies of administrators to fix garbage left behind by non-polite add-ons, or quick fix stuff directly on production environment. Sadly this is not always possible today.

Why uninstall is important?

If you are an add-ons developer and you want to release your code to the community you must live with the possibility that your product will not fit the user's need and he will uninstall it. You must be sure that it will be a simple operation that don't leave problems behind.
If you don't care about this, you will became quickly the Bad Guy/Company and no-one will ever use your work in future. And you will also lose all your friends. And wife. And Cat.

Another fact: a lot of people still "try" products on production sites.
This is happening more frequently of what you can imagine; maybe they started adding stuff to a test or staging site but later this became a production site just because it's now filled by a lot of production data.
We can't simply say "Hey! You didn't read how to evaluate a Plone product guide"! This is a puerile excuse.

Tools for make products uninstall cleanly are around us: just use them.

Today: Generic Setup age

Installation

Before starts talking about product's removal let's ask ourself "how installation works nowadays"?

The portal_quickinstaller (the Plone tool that manage installations) is now Generic Setup compatible: this mean we don't need the magical install.py file anymore.
How? Let me say that there's a well know standard for creating installation Generic Setup profiles: a profile named "default". Probably 99% of add-ons today use this common patter (you can unluckily find some exception, but I think is really uncommon).

So you can think that portal_quickinstaller is looking for a profile named "default", but this is false. The algorithm is more... "simple": it looks for the first profile found (alphabetically sorted) and use this an "installation profile".
When relying on this automatic choice, this will trigger a log warning message:
INFO CMFQuickInstallerTool Multiple extension profiles found for product example.gs. Used profile: example.gs:default

This lead to some considerations:
  • you need to be sure that the first profile found is really the installation profile (so if you want to keep "default" as its name, and I suggest you to do so, you must handle carefully other profiles names).
  • you can't install two profiles from the portal_quickinstaller tool
About the second limitation: Plone spent a lot of energy in last years to free users from the need to access ZMI. When I talk of "using portal_quickinstaller" I mean the Plone UI access to the portal_quickinstaller: the "Add-ons" control panel.

The Add-ons control panel

What is not possible today is load from Plone UI additional setup profiles. This is OK for most products (commonly you simply need an installation profile) but there's some add-ons that bring additional configuration profiles, and for using them they explicitly document the need of still going to ZMI, in the portal_setup tool... not really user friendly.

Trick for having multiple installation profiles for the same product

Ok... I want to be honest.
There's a dirty way for having multiple profiles registered by the same product. We discovered this "feature" recently but I'm not sure if this is something good to teach, so use it at your own risk!

It seems that the portal_quickinstaller tool is loading the first-found profile defined in a Python module, not in a whole product.
The trick: define another Generic Setup profile in a different Python module.

Commonly profiles are defined in the Plone project root, directly inside the configure.zcml file or inside a profiles.zcml file. Let say this file is inside the example.gs module.

So you can include another Python module submodule:

...
  <genericsetup:registerProfile
      name="default"
      title="example.gs"
      directory="profiles/default"
      description="Installs the example.gs package"
      provides="Products.GenericSetup.interfaces.EXTENSION"
      />

  <include package=".foo" />
...
Then inside the example.gs.foo submodule you can register the new profile (with another configure.zcml file).
<configure
    xmlns="http://namespaces.zope.org/zope"
    xmlns:genericsetup="http://namespaces.zope.org/genericsetup"
    i18n_domain="example.gs">

  <genericsetup:registerProfile
      name="foo"
      title="example.gs foo"
      directory="profiles/foo"
      description="Foo profile"
      provides="Products.GenericSetup.interfaces.EXTENSION"
      />
  
</configure>
In this way you'll see another "product" in your portal_quickinstaller tool.

Uninstallation

When you start facing uninstallation, things are more confused.
Plone automatically supports uninstallation of products (this was true also in the pre-Generic-Setup era) performing automatically a lot of stuff we will see later.

The Add-ons control panel

But often, for performing a really good cleanup, you'll need an uninstallation profile.
The Plone standard for this is translated to distribute with your products a profile named "uninstall".
You can expect that portal_quickinstaller tool is looking for some specific profile name (or choosing it with some other fancy algorithm). No!
For being able to perform an explicit uninstallation you must still rely onto the old-friend install.py file.

So: a good Plone add-on today must still be distributed with an Extension folder, with an install.py file.
Inside this file we can skip the "install" function (tip: you can still use it for installing some Generic Setup profile that isn't the first found, or checking the Plone version and keep old Plone compatibility alternative profile) but you must provide the "uninstall" function.
This is how an install.py file can looks like:
def uninstall(portal):
    setup_tool = portal.portal_setup
    setup_tool.runAllImportStepsFromProfile('profile-example.gs:uninstall')

A lost old friend: "reinstall" button

What you can do from the Plone "Add ons" control panel are mainly the same operations you can do from the ZMI, explicitly accessing the portal_quickinstaller tool. But with some differences.

From Plone UI you can simply install (activate) an add-on and uninstall (deactivate) it, but you can also easily perform a product upgrade to newer version when this is needed.

The Add-ons control panel

Products upgrade are a great tools and, when properly used, they helps your users to update your configurations to a newer version.
However sometimes they can be "fragile":
  • when you upgrade from a very old version of the product.
  • when the product's developer didn't work well and he's not providing you an upgrade step.
The Add-ons control panel
In those situations a well-know patter is to uninstall then reinstall the product: this will remove the product's configuration (remember that, if an uninstall profiles is not provided, only some configuration are automatically removed), then a new configuration is done from scratch.

The ZMI view of portal_quickinstaller is different. It only knows about product install and uninstall, totally ignoring the product's upgrade (for this, if you really want to use ZMI, you need to rely on portal_setup tool).
But when a product in installed, you can find another option that the Plone UI is not providing: the "reinstall" button.

The Add-ons control panel
What "reinstall" can do?
Well... if you don't plan something different it's only a shortcut for uninstall+install in sequence, but it can be more.
The install and uninstall functions inside the install.py file can catch "reinstall" action if you change the code like below:
def uninstall(portal, reinstall=False):
    if not reinstall:
        ...
        setup_tool = portal.portal_setup
        setup_tool.runAllImportStepsFromProfile('profile-example.gs:uninstall')
        logger.info("Uninstall done")
So: you can perform different things if the user pressed the "Reinstall" button (commonly: you can perform the uninstallation profile only if don't want to reinstall).

Why this is important? And why IMHO the reinstall button can be something good also in Plone control panel?

When an uninstallation is well done (that's the scope of this article), it's destructive as you really want that everything related to your product is removed from the site: you must be 100% sure that the user can safely remove your code from his buildout without giving him problems.

However sometimes:
  • users are upgrading from a very old version of your product, and the internal migration procedure is not working
  • users want to install again your product, means that they want to fully restore original configuration. User's actions, or other products, can mess up you product configuration.
The only thing that a user can do to restore is... reinstall. But doing a reinstall from Plone UI means performing all the super-clueaning-stuff you have prepared, probably destroying all configuration done by the users. Simply repeating the install profile can be better. 

Let's conclude: sometimes I still like to provide a "reinstall" support.

Handle uninstallation with Generic Setup

Keeping in mind that, the main argument here is products uninstall, let me describe all good things (steps) you can do with Generic Setup and how you must handle those steps when you need to uninstall the product.
For this task, I will describe every know Generic Setup import step and what you need to do (if you need to do something) for uninstall.

All the code below is also accessible in an example product on Plone uninstall.

metadata.xml

The metadata.xml file keeps care of registering the current profile version of the product (from Plone 3.3 and above this is not the product version) but it's doing more: it can run Generic Setup profile dependencies (that are not necessarily egg dependencies).
Roughly speaking: it install products that are dependencies of the add-on you are installing.

<?xml version="1.0"?>
<metadata>
  <version>1000</version>
  <dependencies>
    <dependency>profile-collective.something:default</dependency>
  </dependencies>
</metadata>

This is a double-edged weapon and you can't simply put there all your dependencies: it's better to look at the format of the dependency add-ons.
Why? Sometimes Generic Setup steps are destructive (yes, not only the uninstall profile can damage your configuration): they can reset a piece of configuration, ignoring if the configuration was already present in the site and customized by users.

Image handling settingsSome examples.
  1. If you develop an add-on that is compatible with Plone 3 and Plone 4 and you rely on plone.app.imaging, you must know that this package is not already present on Plone 3: you must add it to your egg dependencies list.
    But adding it also to the Generic Setup dependencies list is not a good idea because running this step will reset the image configuration of the portal (both on Plone 3 and Plone 4).
  2. TinyMCE. if you are developing a TinyMCE & Plone-compatible add-on you can't trigger the TinyMCE Generic Setup configuration on install because this will reset the TinyMCE status of a fresh Plone site.
It this a general plague? No. Some Plone components or add-ons can be putted in the dependencies list safely, but you need to know them (looking at their code).

Two good examples follows.
  1. plone.app.registry, that you can manually install on Plone 3 (with some version pinning work, but still you can do it). Reinstalling it's Generic Setup profile multiple times will do nothing that can be damage your configuration.
  2. A famous Plone add-ons: PloneFormGen.
    This add-on can be enhanced by other Plone add-ons that could add new fields or adapter. This features enhancement is commonly done through adding new types, registered as FormFolder possible subtypes. This is not done relying only onto the types.xml  Generic Setup step (see below), but also through Python scripting.
    In this way you can safely add PloneFormGen to your dependencies list, and reinstall it multiple times without loosing anything special.
    This is a good example of an add-on that want to keep things simple for integrators.

What is missing?

I already discussed this a couple of times in the Plone-Users mailing list but I didn't find other plonistas that share my idea.
Let see what I don't like of this approach: IMHO the dependency check of Generic Setup is lacking of a way to know if the add-ons is already installed (formally speaking: if the Generic Setup configuration has been already executed).
Sometimes (let me say "very often") what you really want is: install a dependency Generic Setup profile only if the product is not installed.
For doing this today: you must rely on Python.

Uninstallation

You don't need to do nothing special to revert changed done by metadata.xml.
After all I said above you can ask yourself "isn't a good attitude to uninstall add-on's dependencies when I remove it?".
Commonly, no. You can't be sure that the one of the dependencies wouldn't be a dependencies of another products.

actions.xml

The actions.xml Generic Setup step is reserved to the configuration of the portal_actions tool you can explore or modify through ZMI. It's a container of CMF Action Categories and CMF Action items, that are used by Plone to draw a lot of interface elements, like links an buttons.

Multiple installations

Repeating the installation of the profile will restore all the actions, as defined in the XML file.

Uninstallation

You must do nothing about actions.xml when you uninstall: Plone will automatically remove all your defined actions.
At first glance this seems a good thing ("wow, I don't need any uninstall step for actions!"), but to be honest it isn't: an explicitly requirements of an uninstall actions.xml on uninstall would be a safer solution.
Right now, if you want too keep an action you defined through Generic Setup when you remove a product, you need to manually re-add it after uninstall.

Managing portal_actions through ZMI is very simple, so I don't find any need of such kind of automatism. Also, most of times, orphan portal_actions elements are safe and simple to be removed.

Personal experience: sometimes customers ask to be able to manually handle portal tabs, as portal tabs are seen by the custom as something content related, not theme related.
If you started adding portal tabs using Generic Setup, the you teach to the customer how to manually change portal tabs using ZMI and later you'll provide to the customer a product version, you need to be sure that theme upgrade will not destroy customer's work.

For this reason, for some kinds of actions categories (like portal tabs), I prefer to not define additional actions using actions.xml but I define them manually using ZMI (or some other add-ons).
For this strange behavior I never liked and for take care of this kind of requests, we developed the collective.portaltabs add-on.

catalog.xml

This is the step able to register new catalog index and metadata:
<?xml version="1.0"?>
<object name="portal_catalog" meta_type="Plone Catalog Tool"> 
   ...
 <index name="getFoo" meta_type="FieldIndex">
  <indexed_attr value="getFoo"/>
 </index>
 ...
 <column value="getFoo"/>
 ...
</object>
Until you use it at install time, it works perfectly.
Problems arise when you re-install a product or if you have already an index registered with the same name in your Plone site: in that case use of index will clean your index values.

For this reason I commonly prefer to register only metadata (column), and use Python to register indexes:
    catalog = getToolByName(portal, 'portal_catalog')
    indexes = catalog.indexes()
    # Specify the indexes you want, with ('index_name', 'index_type')
    wanted = (('getFoo', 'FieldIndex'),
              ('works', 'KeywordIndex'),
              ...
              )

    indexables = []
    for name, meta_type in wanted:
        if name not in indexes:
            catalog.addIndex(name, meta_type)
            indexables.append(name)
            logger.info("Added %s for field %s.", meta_type, name)
    # lines below if you want also reindex items when executing it
    if len(indexables) > 0:
        logger.info("Indexing new indexes %s.", ', '.join(indexables))
        catalog.manage_reindexIndex(ids=indexables)

Multiple installations

Repeating the Generic Setup configuration will register again the index configuration. A side effect of this is the index cleanup. This will force you to perform a reindex.

Instead, metadata columns are kept without loosing any data.

Uninstallation

When you uninstall a product, no changes are done to the Plone catalog: you must handle the indexes and columns uninstallation yourself:
<?xml version="1.0"?>
<object name="portal_catalog" meta_type="Plone Catalog Tool">
 <index name="Creator" meta_type="FieldIndex" remove="True">
  <indexed_attr value="Creator"/>
 </index>
 <column value="Creator" remove="True"/>
</object>
Indexes and metadata left behind will not give you any trouble, however cleaning the catalog from useless elements will quick-up the indexing operations.

componentregistry.xml

This is the place where sometimes Plone can gives big headache.
Component registry is a powerful tool that you can use to register local utilities, adapters and subscribers. This section will continue talking of utilities (that are most common components used there)
After the ZCA utility registration, you can register the utility in the component registry of your site on product install, using Generic Setup and the componentregistry.xml file:
<?xml version="1.0"?>
<componentregistry>
  <utilities>
    <utility
      interface="example.gs.interfaces.IFooUtility"
      factory="example.gs.utility.FooUtility"
    />
  </utilities>
</componentregistry>
Syntax above can change if the utility is a named utility and this is often used with a Plone tool registration adding the object attribute (see toolset.xml below for Plone tool info).

Uninstallation

What can happen if you don't provide an uninstallation profile?
Things may differ: sometimes you'll get immediately an ugly ZMI error as soon as you remove the package from the buildout:
TypeError: ('object.__new__(FooUtility) is not safe, use Persistence.Persistent.__new__()',
<function _dt_reconstructor at 0x110df1a28>,
(<class 'example.gs.utility.FooUtility'>, <type 'object'>, None))
Other times the problem stay under the hood for weeks, months... I started this article talking of Plone site migrations to new a version: this is exactly the perfect moment where such kind of problem arises and you find that a very old utility, of a product you badly removed ages ago, now gives you problems.

Can we do something from ZMI? Do you remember when I said that ZCA is a recent technology?
Well... You have a ZMI tab ("Components") on every Plone site, where you can see registered stuff.
Site Components tab
The UI seems editable, but I never had luck using this: on my past tests it never really cleanup things and now (Plone 4.2.4) I get a traceback error.
So: use it for seeing what you have registered, but nothing more.

So: how to uninstall?
First of all a good uninstall step:
<?xml version="1.0"?>
<componentregistry>
  <utilities>
    <utility
      interface="example.gs.interfaces.IFooUtility"
      factory="example.gs.utility.FooUtility"
      remove="True"
    />
  </utilities>
</componentregistry>
After running this, the "Components" tab view above seems cleaned but still, if you remove the egg from your buildout you get same problems.
This is orrible. Nowadays there's no way of really cleaning components using Generic Setup or ZMI.

So: you need to switch to Python!
There's a great article about how to manually removing local persistent utilities. This document is also a good starting point for removing other persistent stuff like adapters and subscribers.
The core of the approach is usage of getSiteManager() method:
>>> sm = app.Plone.getSiteManager()
>>> sm.utilities
<zope.component.persistentregistry.PersistentAdapterRegistry object at 0x1096b9c80>
>>> sm.adapters
<zope.component.persistentregistry.PersistentAdapterRegistry object at 0x1096bd1b8>
>>> sm.subscribers
<bound method PersistentComponents.subscribers of <PersistentComponents /Plone>>
Now, let's see how to remove our dummy utility registered above:
def _removePersistentUtility(portal):
    sm = portal.getSiteManager()
    sm.unregisterUtility(provided=IFooUtility)
    util = sm.getUtility(IFooUtility)
    del util
    sm.utilities.unsubscribe((), IFooUtility)
    del sm.utilities.__dict__['_provided'][IFooUtility]
    logger.info("Removed utility")

def uninstall(portal, reinstall=False):
    if not reinstall:
        setup_tool = portal.portal_setup
        setup_tool.runAllImportStepsFromProfile('profile-example.gs:uninstall')
        _removePersistentUtility(portal)
        logger.info("Uninstall done")
But how can you clean this if you are facing this kind of issue with a product that you haven't developed yourself (the Evil Prone Developer)?
There's a Plone add-ons that is often used in such kind of situation: it is wildcard.fixpersistentutilities. It's performing tasks above giving you a good Zope user interface, making simple finding broken persistent garbage.
Use it you you really need it, but please... don't pretend that your users will use it for uninstalling your product: provide an uninstall procedure yourself!

controlpanel.xml

This is the step where you can register new Plone control panel link to your product's configuration section.
<?xml version="1.0"?>
<object name="portal_controlpanel"
        xmlns:i18n="http://xml.zope.org/namespaces/i18n" i18n:domain="example.gs">
    <configlet title="Example configuration"
        action_id="Example" appId="example.gs"
        category="Products"
        condition_expr=""
        icon_expr="string:$portal_url/maintenance_icon.png"
        url_expr="string:${portal_url}/@@example-configuration"
        visible="True"
        i18n:attributes="title">
        <permission>Manage portal</permission>
    </configlet>
</object>

Uninstallation

The story of control panel uninstallation is full of products that leave broken link behind. The broken link is not something that can brake your site, but it's really annoying to see them around.

Uninstall is simple:
<?xml version="1.0"?>
<object name="portal_controlpanel"
        xmlns:i18n="http://xml.zope.org/namespaces/i18n" i18n:domain="example.gs">
    <configlet title="Example configuration"
        action_id="Example" appId="example.gs"
        remove="True">
        <permission>Manage portal</permission>
    </configlet>
</object>
Recently (not sure from which Plone version) the control panel can be cleaned also from ZMI, using directly the portal_controlpanel tool.
portal_controlpanel tool But as you seen, providing an automatic uninstall of the control panel link is very easy. Do it!

cssregistry.xml

The tool that takes care of managing all Plone CSS, merging and minimizing them: portal_css.
Every products that want to register a CSS in the tool must provide one or more snippet like this:
<?xml version="1.0"?>
<object name="portal_css">
 <stylesheet title=""
    id="++resource++example.css"
    media="all" rel="stylesheet" rendering="import"
    cacheable="True" compression="safe" cookable="True"
    enabled="1" expression=""/>

 <!-- more follow -->
</object>
From now, the portal_css tool will try to load the example.css recourse.

Uninstallation

What's happen if the resource is not found, because you removed the product but you didn't clean the tool?
The portal_css tool is configurable manually, and it's take care of warning the user about missing resources, so you can clean manually.
portal_css tool
Automatic cleaning on uninstall is simple:
<?xml version="1.0"?>
<object name="portal_css">
 <stylesheet id="++resource++example.css"
    remove="True" />
</object>

factorytool.xml and types.xml

I'll talk about those two kind of Generic Setup steps because they are used at the same time and they have same behaviours. They manage Plone content types.
The factorytool.xml file is configuring the ZMI portal_factory tool and is used only for Archetypes based content types.
The types.xml file (and the types directory and XML sub-elements) are used for registering and managing the content type itself (for both Archetypes and Dexterity based content types) through the portal_types tool.

I don't find useful to provide here some XML snippet because all this stuff is commonly generated automatically for you using paster (or ZopeSkel, or plone.templter, or mr.bob... whatever tool you are using).

Multiple installations

Repeating a type installation will totally restore the starting configuration, cleaning all the administrator manual changes.

Sometimes additional products would try to touch only some part of a type configuration. For example: you need to add a new available view to the Folder type (views are controlled by the view_methods property).
You can use three ways:
  • The Bad Ones. You will register the Folder type from scratch, providing the modified view_methods.
    Not good: you will clear all configuration changes done by the site administrator.
  • The Still Bad Ones. You can provide an empty type definition with just the modified view_methods.
    Not too bad, but you will clean all configuration done by others to the list of views.
  • The Good Ones. Provide only the view_methods (like above) with the single new entry, and adding the 'purge="False"' attribute.
    This will only add your new view.

Uninstallation

You don't need to do anything! Sometimes I found  Plone add-ons that provide an uninstall XML for those two steps, but they are ignored.
Those two tools are of the type that automatically handle uninstallation: without doing anything it remove configuration when uninstalling.
Unlike the same magical behavior we saw for the portal_action tool, this time I like this kind of automation (and I've never had problems with it).

What you must note is that you removed a type definition. What happens to existing content of the type that lives in the site?
Well... as soon as you'll remove the type code from the buildout, the will be broken.

So: can be a good idea to automatically crawl the site and remove all content of this type?
I think no... and I don't know any product that is doing this. If your product user uninstalled the product by mistake you can waste a lot of work (well... you can hope that undo support will work)!

jsregistry.xml

This import step manage the portal_javascript tool, for Plone JavaScript resources.
There's nothing I can say about this tool that I didn't reported about the cssregistry.xml import step, so I'll be quick.

The installation process is similar to the ones about style sheets:
<?xml version="1.0"?>
<object name="portal_javascripts" meta_type="JavaScripts Registry">
 <javascript cacheable="True"
             compression="safe"
             cookable="True"
             enabled="True"
             id="++resource++example.js"
             inline="False"/>

 <!-- more follow -->
</object>

Uninstallation

The ZMI tool can be used for manually removing old registrations:
portal_css tool But clean uninstallation is always simple:
<?xml version="1.0"?>
<object name="portal_javascripts" meta_type="JavaScripts Registry">
 <javascript id="++resource++example.js"
             remove="True"/>
</object>

memberdata_properties.xml

It's not so common, but sometimes additional products need to add new Plone users data in the portal_memberdata tool. This tool can also be accessed and modified manually through ZMI.
Recent Plone versions switched to plone.app.users module, so the importance of this tool is decreased.

Registering new properties is simple:
<?xml version="1.0"?>
<object name="portal_memberdata">
 <property name="foo" type="string"></property>
 <!-- other can follow -->
</object>

Uninstallation

There's no Generic Setup support, so you need to use Python.
...
def _removeUserProperty(portal):
    portal.portal_memberdata.manage_delProperties(['foo'])
    logger.info("User property removed")

def uninstall(portal, reinstall=False):
    if not reinstall:
        ...
        _removeUserProperty(portal)
        ...

portal_atct.xml

This Generic Setup import step is used to configure the portal_atct (ATContentTypes) tool, related to some features of the current Plone content types family.
It can be splitted in two kind of configurations.

The first section in related to collections criteria definitions but only for old Plone collection (that are disabled starting from Plone 4.2, which now rely onto plone.app.collection).

The other section configure some general property like image_types, folder_types, album_batch_size, album_image_scale and single_image_scale. To be honest I'm not sure if they are still used nowadays.

So, let's focus on collections only.

The strange thing about this tool is that you can't manage collections configuration using ZMI but only from Plone UI, so you can manually configure (changing properties, or disable) collection criteria or columns (used for the "collection view") calling the "/portal_atct/atct_manageTopicIndex" view.
portal_css tool What you can't do is to add new new criteria or columns, or delete old ones.

To add new criteria or columns you need a simple Generic Setup XML:
<?xml version="1.0"?>
<atcttool xmlns:i18n="http://xml.zope.org/namespaces/i18n">
 <topic_indexes>
  <index name="getFoo"
     description="A really dumb field" enabled="True"
     friendlyName="Foo">
   <criteria>ATSimpleStringCriterion</criteria>
   <criteria>ATListCriterion</criteria>
  </index>
  <!-- more can follow -->
 </topic_indexes>
 <topic_metadata>
  <metadata name="getFoo"
     description="A really dumb field"
     enabled="True" friendlyName="Foo" />
  <!-- more can follow -->
 </topic_metadata>
</atcttool>

Uninstallation

Old style collection criteria are based on catalog indexes. If the catalog index used by the criteria disappear, the "/criterion_edit_form" view will be broken.
Sadly, as sayd above, you can't cleanup things and delete the criteria from the configuration form. Wow! Now all your old-style collections are broken!
What you can do is to disable the criteria; in this way collections would work again, but for a real cleanup, you must provide an uninstall step:
<?xml version="1.0"?>
<atcttool xmlns:i18n="http://xml.zope.org/namespaces/i18n">
 <topic_indexes>
  <index name="getFoo"
         remove="True">
  </index>
  <!-- more can follow -->
 </topic_indexes>
 <topic_metadata>
  <metadata name="getFoo"
            remove="True"/>
  <!-- more can follow -->
 </topic_metadata>
</atcttool>

portlets.xml

There's a lot of things we can say about portlets in Plone but I already dealt with this argument in another article:  "Plone portlets are not enemies... just rude friends". However the time of this old article something's changed.

Using the portlets.xml file you can register new portlet manager and portlet.
Registering new portlet manager is not common, while registering new portlet is a natural part of Plone add-ons world.
<?xml version="1.0"?>
<portlets xmlns:i18n="http://xml.zope.org/namespaces/i18n">
   <portlet\
     addview="example.gs.portlet.Example"
     title="Example portlet"
     description="An example portlet"
   />
</portlets>
Apart the portlet registration (so: seeing the new portlet available in the "add new portlet menu") there's also the persistent object you will create on the database (the portlet assignment).
This is, in fact, the most tricky part.

Last note: again, portlets are a recent Plone addition so you can't do absolutely anything from the ZMI for using/fixing/cleaning them.

Uninstallation

As already told in the other article, uninstalling the product that register the new portlet is enough for preventing Plone to raise errors and making new portlet to disappear from the add menu, but the real problem are the persistent instance created, that must be deleted before.
Good news: there are recent changes about it: ticket #7375 has been closed, so sooner will be possible deleting broken portlets from Plone (EDIT: seem already possible in Plone 4.2.5). There are also other experimental products.

While writing this article, I'm also looking at all importexport modules of Generic Setup world. Looking at portlet code you can see that an uninstall procedure is available (see the exportimport/portlet.py source).
A docstring says:
Remove a portlet definition using the 'remove' attribute so that it can no longer be added via @@manage-portlets. This does not remove any assignments:
<portlet remove="True" addview="portlets.Calendar"/>
To be honest, I never used a this remove attribute and I didn't get any errors in years, so probably this can be used to removed/unregister existing portlets you don't want to show to your users (not bad if you plan to substitute an existing portlet with a new ones).

Looking at the same code, there's also a purge feature. The docstring of the corresponding method says:
Unregister all portlet managers and portlet types, and remove portlets assigned to the site root
Great! This will introduce the last problem of uninstalling products with portlets: although is now possible to delete broken portlets, I think that a good uninstall must take care of cleaning portlets assignments left behind in the site.

The purge feature above seems a good starting point, however is not enough; removing portlets in the site root can be only the edge of the iceberg: portlets in Plone can be added everywhere, on every content and there's not simple way to know where.
The only thing you can for cleaning all contents is to traverse all your site tree and check every content... this is not something you can do when uninstalling: it can take hours and slow down your site.
What is missing? The problem is that Plone is not memoizing where you added new portlets (probably a marker interface would be enough), so you can't quickly identify those contents.

propertiestool.xml

The Plone portal_properties tool is a well-know, robust component of the Plone world. It's a composition of "Plone Property Sheet" objects, that store a set of configuration (of primitive types, like strings, integers float, boolean, lists, ...). portal_css tool In the past this was the only place where you could add configuration for your products, and the Plone core itself was using this. Also today, a lot of configuration are still there.

What I always liked of properties sheets is that you easily managing them from ZMI, and cleaning garbage is simple. Also, APIs for using them are really simple. portal_css tool However I need to say that Plone configuration are now quickly moving to the Plone registry (see the registry.xml section below).

Registering new properties or a while new property sheets is really simple:
<?xml version="1.0"?>
<object name="portal_properties">
 <object name="navtree_properties" meta_type="Plone Property Sheet">
  <property name="metaTypesNotToList" type="lines" purge="False">
   <element value="ExampleType"/>
  </property>
 </object>
 <object name="example_properties">
  <property name="title">totally useless properties</property>
  <property name="foo" type="boolean">True</property>
  <property name="bar" type="string">xxx</property>
 </object>
</object>
You can use this tool for changing default Plone configuration (note the use of purge option above, for not cleaning the whole metaTypesNotToList property) and registering your new property sheet.

Property sheets are doing nothing by their own: your product must use them fro something.

Multiple installations

Multiple installation of a propertiestool.xml profile will restore every time the configuration. If the site administrator had manually changed something in the configuration, or a 3rd part product did it, you are probably deleting those configurations.

For this reason I commonly prefer using Generic Setup for adding new property sheets, but still using Python for adding properties.
In this way you can check if the property already exist, and skip it:
_PROPERTIES = [
    dict(name='foo', type_='boolean', value=True),
    dict(name='bar', type_='string', value='xxx'),
]

def registerProperties(context):
    ptool = getToolByName(context, 'portal_properties')
    props = ptool.example_properties
    
    for prop in _PROPERTIES:
        if not props.hasProperty(prop['name']):
            props.manage_addProperty(prop['name'], prop['value'], prop['type_'])
            logger.info("Added missing %s property" % prop['name'])
        else:
            logger.info("Property %s already present. Skipping" % prop['name'])

def setupVarious(context):
    if context.readDataFile('example.gs_various.txt') is None:
        return

    portal = context.getSite()
    ...
    registerProperties(portal)
The plone.app.imaging problem example used at the beginning of this article is, in facts, related to property sheets registration. 

Uninstallation

There's no Generic Setup support for cleaning portal_properties tool, you must provide your Python code. However property sheets left behind are really simple to clean using ZMI (and they don't create any issues if you don't remove them).

Real question is: what to clean?
If you added a totally new Plone Property Sheet, answer is simple: let's remove it.
def _removeProperties(portal):
    portal.portal_properties.manage_delObjects(['example_properties'])
    logger.info("Property sheet removed")

def uninstall(portal, reinstall=False):
    if not reinstall:
        setup_tool = portal.portal_setup
        setup_tool.runAllImportStepsFromProfile('profile-example.gs:uninstall')
        ...
        _removeProperties(portal)
        logger.info("Uninstall done")
If you changes some other Plone default properties you must be careful and be sure of what you are doing.

rolemap.xml

The rolemap.xml file can be used to customize new permissions defined by your product, or change some default site permissions.

Uninstallation

There's no Generic Setup uninstall support for this, and in both cases introduced above, this is probably good.

If you are removing a product that add a new custom permission, the new defined permission will not be used anymore (and probably the real question will be "how to delete a Plone custom permission?").

In the case you have customized a 3rd party permission, you could provide a rolemap.xml file that will try to restore the old permission configuration.
An example: you create a policy product that gave the "Review portal content" permission also to Editor role. On uninstall you can provide a new permission map that restore the starting Plone configuration.
However this is not a 100% secure choice and the reason is always the same: can you be sure that no 3dr party Plone products or administrator manual changes touched this permission? You can't.
Unluckily the rolemap.xml file doesn't provide a 'purge="False"' feature (for editing only the permission for a single role).

Again: evaluate what to do.

skins.xml

Probably 50% of garbage you can see in a Plone site after removing a product came from this import step!
The portal_skins tool is the home for old style CMF technology and old-style Plone themes but is still used today for a lot of Plone feature and add-on products.

From this tool you can define new plone skins and modify layers set of existing skins.

To register a new layer and new skin (common style for Plone theme add-ons):
<?xml version="1.0"?>
<object name="portal_skins">

 <object name="example-gs"
    meta_type="Filesystem Directory View"
    directory="example.gs:skins/example-gs"/>
 <skin-path name="example-theme" based-on="Sunburst Theme">
  <layer name="example-gs"
     insert-after="custom"/>
 </skin-path>
</object>
For register only a new layer for all existing skins (common style for classic add-ons that need CMF skins resources):
<?xml version="1.0"?>
<object name="portal_skins">
 <object name="example-gs"
    meta_type="Filesystem Directory View"
    directory="example.gs:skins/example-gs"/>
 <skin-path name="*">
  <layer name="example-gs"
     insert-after="custom"/>
 </skin-path>
</object>
Both registration are splitted in two section: the first one is registering a filesystem directory as a possible additional layer, but this will not be used without the second section.

Uninstallation

Items inside portal_skin tool are easy to manage and clean through ZMI and broken stuff left behind by lazy products will not give you any problem.
However layers and skins removal is simple.

First of all: you only need to care about unregister the skin-path. My tests reported that the filesystem directory registration is automatically removed when you uninstall the product.

For totally remove a theme:
<?xml version="1.0"?>
<object name="portal_skins">
 <skin-path name="example-theme" based-on="Sunburst Theme" remove="True">
  <layer name="example-gs"
     insert-after="custom"/>
 </skin-path>
</object>
For remove only a layer (from all skins where it has been registered):
<?xml version="1.0"?>
<object name="portal_skins">
 <skin-path name="*">
  <layer name="example-gs" remove="True" />
 </skin-path>
</object>

toolset.xml

Portal tools are singleton, persistent items stored in the site root, visible from ZMI and (commonly) invisible from Plone interface. For a long time, portal tools has been the only place where Plone developers were able to store data and code together.

When ZCA and persistent utility were integrated this kind of items has been partially deprecated. Today there's still a lot of Plone tools, but commonly they are also register as persistent component.

Tools are very simple items, and the Generic Setup support is limited only to the tool registration:
<?xml version="1.0"?>
<tool-setup>
  <required tool_id="portal_foo"
            class="example.gs.tool.FooTool"/>
</tool-setup>

Uninstallation

There's not Generic Setup supporto for deleting tools. If you remove the product from your buildout you'll see a broken object in the site root (because the code is no more available), but it's really simple to delete it using ZMI.

To automatically delete it, you can use Python, but one time again take care on users that are reinstalling your product: if you delete the tool you will lose all configuration stored inside.

Obviously, if the tool is also registered in the component registry, you need to take care of it (see the componentregistry.xml section above).
def _removeTool(portal):
    portal.manage_delObjects(['portal_foo'])
    logger.info("Portal tool removed")

def uninstall(portal, reinstall=False):
    if not reinstall:
        setup_tool = portal.portal_setup
        ...
        _removeTool(portal)
        logger.info("Uninstall done")

viewlets.xml

Not much to say about that, just because it is mainly used by Plone themes, to:
  • Move viewlets around (this is only for theme... for sure)
  • Hide existings viewlet (this can be done because you are providing a new viewlet that replace a default ones)
Note that all this can be done through the Plone @@manage-viewlets view.
I'm not an expert of this import step, but now you can find a good viewlet Generic Setup documentation in the code itself.

Uninstallation

An uninstallation Generic Setup step for viewlet could be really similar to an installation one: you only move viewlets and probably un-hide hidden ones (that mean "remove the hidden state"...).

What you must do when somebody uninstall your theme? You must restore viewlet in the original order? And what is the original order? How can you know it?
Probably you can't do much (apart showing viewlets you hidden).

workflows.xml

Workflow is a complex argument, and also the Generic Setup syntax can be complex.
It can be splitted in two section:
  • Defining workflow (state, transitions, ...)
  • Defining bindings of workflows to Plone types
Workflow can contains code, but most of the times the code is inside the workflow itself, so totally stored in the site database: this mean that you can remove the product who defined a workflow and still get it working.

Uninstallation

Workflow are stored and manageable through the portal_workflow ZMI tool, so is quite easy to perform manual cleanup.

How embarrassing! I'm obviously focused on understand how to remove created workflow but this time I'm not able to find where Plone is managing Generic Setup for this (I'm missing the type bindings part).
So I can't be sure if you can remove workflow items using Generic Setup or not., but I suspect that there isn't any Generic Setup support for uninstallation.

What is important to know is if you need to uninstall (and what).
Is a good idea to remove a workflow that your product installed? Can you be sure that another products, or the site manager is not using it anymore for other types?
This is not so impossible: while defining new workflow is done using Generic Setup or ZMI, the workflow associations to types can be done (and is really recommended) using the Plone control panel ("Types configuration").

Be aware of that.

import_steps.xml

Historically some old version of paster and probably old documentation generated a lot of Plone add-ons where additional operations to be done with Generic Setup import where stored inside code executed using import_steps.xml:

The use of this Generic Setup step can be used to register a new possible step inside the portal_setup tool.
An import steps example
The new way of getting this kind of features is to use ZCML, and genericsetup:importStep:
<genericsetup:importStep
      name="example.gs.various"
      title="example.gs: miscellaneous import steps"
      description="Various import steps that are not handled by GS import/export handlers."
      handler="example.gs.setuphandlers.setupVarious">
      <depends name="propertiestool"/>
</genericsetup:importStep>

Uninstallation

When the registered code disappear, the import step will be broken.
An import steps example
This kind of errors (that only trigger some log warning messages) can be easily removed from the "Manage" tab of the portal_setup tool where you'll find a "Steps with invalid step handlers" section.

Like always, you can use Python to automatically fix things:
...
def _removeImportStep(portal):
    to_remove = ('example.gs',)
    registry = portal.portal_setup.getImportStepRegistry()
    for step in to_remove:
        if step in registry._registered:
            registry.unregisterStep(step)
            logger.info("Removing import_step %s" % step)

def uninstall(portal, reinstall=False):
    if not reinstall:
        setup_tool = portal.portal_setup
        setup_tool.runAllImportStepsFromProfile('profile-example.gs:uninstall')
        ...
        _removeImportStep(portal)
        logger.info("Uninstall done")

plone.app.registry.xml

Plone 4 included a new component: the Plone registry. The idea behind this is really good: simply speaking this new component can replace the old portal_properties tool (see above) with something that is:
  • more powerful
  • data model can be defined using interfaces
  • usable from Plone UI (there's not ZMI support for it)
IMO the target has been partially reached: the old portal_properties tool is still more robust the Plone registry, and can be easily cleaned up (but not so powerful).

Somethings things goes worse: let's talk of new style collections.
If you are providing new collection criteria in one of you product add-on, you are also registering items in the registry using the plone.app.querystring prefix, that is not present on Plone 4.1 and below: so you are breaking Plone 4.1 compatibility.

Don't want to repeat myself: I wrote another article about Plone registry on RedTurtle Blog.

Uninstallation

Again: don't want to repeat myself: in a second article about Plone registry I introduced how to uninstall properly and manage registry changes in the data model.

About new collection criteria: things are not perfect. Removing registered criteria from the registry partially works only on Plone 4.3...
Probably is possible to do something using Python, but right now I've no idea.

browserlayer.xml

Browser layer is a really powerful Plone toy: it make possible to register views (or customize existings views) only when your product is installed.
Here how to do this with Generic Setup:
<?xml version="1.0"?>
<layers>
    <layer name="example.gs"
           interface="example.gs.interfaces.IExampleGSLayer" />
</layers>

Uninstallation

For a long time I didn't provided an uninstall step for browser layers, and you'll find a lot of products that are ignoring this. When you remove a product that registered a browser layer from your buildout, sometimes you don't get any problem.

I started getting a lot of problem with this on Plone version migration.

The truth: you must give an uninstall step for this, and Generic Setup is supporting it:
<?xml version="1.0"?>
<layers>
    <layer name="example.gs"
           interface="example.gs.interfaces.IExampleGSLayer"
           remove="True" />
</layers>

Other less used Generic Setup steps

There are other Generic Setup import steps like diff_tool.xml, contentyperegistry.xml, kss_registry.xml (KSS has been removed from Plone core starting with Plone 4.3), ... it's quite uncommon to use them so I don't spent time here to provide examples, and they commonly don't provide uninstall procedure.

Also not that 3rd part product could provide additional Generic Setup support for other components: an example is ATVocabularyManager.

Conclusion

A single rule: for every thing you install with your product, provide an uninstall counterpart:
  • Your users will love it
  • You'll love it when you will need to remove or change your product in future, for example while migrating to new Plone versions.
Take a look at example.gs.