A short guideline on how to migrate from CKAN 2.8 to CKAN 2.9.
12 Oct 2022
This is a short guideline on how to migrate from CKAN 2.8 to CKAN 2.9. For those of you who are learning about CKAN now, it is a tool for creating open data websites. It helps you manage and publish collections of data. It is used by national and local governments, research institutions, and other organizations that collect a lot of data. Now, let`s take a look at the process of migration.
How to migrate from CKAN 2.8.9 to CKAN 2.9.4
As is with any open-source project, the first place to check is the community for already migrated extensions from the upstream repositories. If an extension was already migrated, it can be pulled from upstream and just merged with custom HTMLs and the translations for the extension. For the extensions that were not migrated, there is this general blueprint that can be followed, tackling the migration one extension at a time:
Migrate Python 2 to Python 3
Migrate the web framework Pylons to Flask
Integration of the key CKAN dependencies
Upgrade of the UI in order to be compatible with the changes done through the migration of the web framework Pylons to Flask
Migrate the current tests from nosetests to pytests
Migrating Python 2 to Python 3 is the simplest task, just simply change python 2 specific syntax to python 3. After this, there are some Python 3 compatibility issues for example bytes vs str handling, unicode, use of basestring, relative imports etc, but these errors can be handled while working on the next steps.
The next three steps are interlinked. With these steps comes code modernization. First, check whether any python libraries that the extension depended on are Python 2 only. If that is the case, you should find suitable replacements or updates. A useful tool for this is the caniusepython3 library, which can help in identifying dependencies that can’t run on Python 3. After this, comes the migration of all the controllers to flask views/blueprints. This means changing all BaseController controllers handled by the IRoutes interface to blueprints handled by IBlueprint interface. Simply put, the code went from looking like this:
import ckan.plugins as p
def before_map(self, _map):
# New route to custom action
# Overriding a core route
to looking like this:
import ckan.plugins as p
blueprint = Blueprint('foo', self.__module__)
rules = [
('/foo', 'custom_action', custom_action),
('/group', 'group_index', custom_group_index), ]
for rule in rules:
If a plugin registered CLI commands, it should be migrated from paste command to Click CLI commands. These migrated commands are integrated into the existing ecosystem via the new IClick interface.
For migrating the frontend resources, the resources registered in resource.config or the files in the fantastic location such as ckanext/ext/fanstatic/script.js and ckanext/ext/fanstatic/style.css had to be loaded into any HTML through webassets.yaml. An example web assets the file looks like this:
ext_script: # name of the asset
filters: rjsmin # preprocessor for files. Optional
output: ckanext-ext/extension.js # actual path, where the generated file will be stored
contents: # list of files that are included in the asset
ext_style: # name of asset
output: ckanext-ext/extension.css # actual path, where generated file will be stored
contents: # list of files that are included into asset
In the case of fantastic it is possible to load CSS and JS files into one asset. For web assets, those files need to be split into two separate assets. So, after migration to webassets.yaml, the templates from loading resources are updated to assets.
After this step is completed, there is a functioning extension and with the final step of re-writing the tests, you can be sure that there were no new bugs introduced in the process. The tests were re-written from nosetests to pytests which meant removing imports from nose.tools to import pytest. The assert statements also had to be changed:
assert_raises(exc, func, *args)
with assert_raises(exc) as cm:
assert 'error msg' in cm.exception
assert x == y
assert x == y
assert x in y
with pytest.raises(RuntimeError) as excinfo:
assert 'error msg' in str(excinfo.value)
The old FunctionalTestBase was replaced by various pytest fixtures. These fixtures are decorators that were added directly to the test class or method. After this last step of migrating the tests was complete, the whole migration of the extension was completed.
Migrating from CKAN 2.8 to CKAN 2.9 is an interesting journey! If you have any questions about the migration process, you can always reach out to us here.
Note: This post was written by Hristijan Vilos, Software Developer at Keitaro. You can see the original post on their blog.
Welcome to CKAN Monthly Newsletter #1! It's been a while since we last connected! We're thrilled to bring you the first edition of our monthly newsletter! We believe that this newsletter will become an essential source of information for our community, and it's going to evolve and improve every month.