CKAN 2023 - A year of milestones

Intro

2023 was a landmark year for CKAN, marked by global recognition, innovative projects, pivotal developments, collaborations, and initiatives that have collectively advanced CKAN's mission of fostering open data and empowering communities worldwide.

CKAN: More Than a Platform, A Philosophy

In the vast expanse of the digital world, CKAN stands tall not just as a platform, but as a philosophy. It’s a declaration that data should not be caged. It should flow freely, accessible to all, unbound, and transformative. This vision is brought to life across a spectrum of applications of CKAN, from traditional open data portals to customized solutions serving businesses, NGOs, and the scientific community.

CKAN’s footprint is massive, spanning hundreds of data portals worldwide (a nod to the recent POSE project's discovery of 381 active CKAN data portals in 59 countries). These portals include the crème de la crème of open data portals – from the EU, USA, UK, Canada, Switzerland, Finland, Argentina, Japan, Germany, etc., to the vibrant shores of UAE, Singapore, and beyond. CKAN’s essence transcends data storage. It’s a beacon of accessibility, a cornerstone of government transparency, and a catalyst for informed data-driven decisions.

CKAN reached a significant milestone, being recognized as a Digital Public Good by the Digital Public Goods Alliance. This is a testament to CKAN’s contribution towards the United Nations' Sustainable Development Goals. It reinforces our role in global initiatives, cementing our place as a vital tool in the pursuit of global development and humanitarian efforts.

CKAN's role has evolved far beyond that of a mere data management platform. It has become a symbol of open data, a bridge connecting diverse communities, and a tool for ensuring transparency and accountability. Our journey continues with a clear vision: to democratize data access and build a world that's more open, more informed, and deeply connected.

As we celebrate our achievements, let’s recap the milestones that have marked this extraordinary year:

Digital Public Good Recognition

CKAN is now a Digital Public Good (DPG). The Digital Public Good Alliance approved Open Knowledge’s application and has added it to the Digital Public Good Registry. The goal of the DPGA and its registry is to promote digital public goods in order to create a more equitable world. Being recognized as a DPG increases the visibility, support for, and prominence of open projects that have the potential to help attain the Sustainable Development Goals (SDGs) put forward by the United Nations (UN). To become a digital public good, all projects must meet the DPG Standard to ensure that projects truly encapsulate open-source principles. by the Digital Public Good Alliance, enhancing its global visibility and reinforcing its role in achieving the UN's Sustainable Development Goals. This recognition underscores CKAN's influence in addressing global challenges and advancing societal progress through open data​​.

Read more here: CKAN is now a Digital Public Good

CKAN 3.0 Taskforce

An important development was the formation of the CKAN 3.0 Taskforce, dedicated to working on the next major version of CKAN. Building on the research and initiatives initiated last year, this year marked a significant leap from conceptualization to action. This initiative has been bolstered by the addition of Dragan Avramovic from Datopian and Brett Jones from Link Digital, bringing additional expertise to the table. Collaborating closely with technical lead Anuar and product owner Alex Gostev, the CKAN Taskforce 3.0 is actively shaping the future of CKAN. Their collaboration is further enriched by the involvement of co-steward Steven De Costa. Their work reflects CKAN Project's dedication to continual growth and its adaptability in meeting the evolving needs of a diverse user base.

Here’s what the CKAN Taskforce achieved in 2023:

• 12 solution and 6 problem hypotheses were formulated and vigorously discussed. They became a foundation for collaborative efforts for task refinement and prioritization. So we have mapped future activities in the form of agreements between members of the community.
• Dragan was added to the team. He made several in-depth technical researches that were presented during core dev team meetings:
  • Lossless harvesting storage with 2 databases (graph and postgres).
  • Making CKAN Modular to Accommodate Various Search Engines in Place of SOLR.
  • CKAN + Triplestore.
• Dragan works right now to extend the search capabilities of CKAN.
• Patricio Del Boca delivered pilot integration of HTMX which was warmly welcomed by the core dev team members and is an important step in up-leveling CKAN to a modern tech stack.
• Brett Jones is doing a great job of supporting newcomers and producing great instructional videos on how to start working with CKAN.
• UI Revamp project to update out-of-the-box UI of CKAN is agreed upon between community members and prioritized so that the UI Revamp team is in the formation stage. This endeavor is made possible with the valuable assistance and sponsorship of the two co-stewards, Datopian and Link Digital. Stay tuned for further updates as this project progresses.

For an in-depth view of the ongoing progress and to join us on this exciting journey, explore the following resources:

• CKAN 3.0 Taskforce GitHub Board: Track the real-time developments and collaborative work shaping the future of CKAN.
• CKAN Future Miro board: Outcome-Opportunity-Solution trees mapped for GitHub discussion we had.

Release of CKAN 2.10

A major highlight of the year was the release of CKAN 2.10. This release is the culmination of a huge effort by all the project contributors and brings a long list of new features and enhancements to make CKAN more secure and easier to customize. Read more here: The latest CKAN release is here, say hi to CKAN 2.10!

Expanding CKAN's Knowledge Base

In 2023, Brett Jones, dedicated his expertise to enhancing the CKAN community's technical prowess, creating 10 new in-depth walkthroughs. Looking ahead to 2024, we are excited to announce that Sergey Motornyuk will be continuing this initiative. More than 15 new videos are slated for release, covering a wide range of topics across the entire CKAN stack. These tutorials are designed to help users skill up and gain a deeper understanding of CKAN's capabilities. Some of the key topics covered include:

• CKAN Extensions: Install into a new CKAN docker development environment
• CKAN Extensions: Updating templates
• CKAN Extensions: Using the CKAN Plugins toolkit
• Watch more tutorials

POSE Project's Contribution to the CKAN Ecosystem

The POSE Project (Pathways to Enable Open-Source Ecosystems), a collaboration between the University of Pittsburgh, datHere, and the U.S. National Science Foundation, has made significant contributions to the CKAN ecosystem. Through a series of expert-led workshops, the project has gathered invaluable insights and strategies aimed at strengthening CKAN's role as a leading platform for open data, government, and science​​​​. The collaboration led to ecosystem discovery activities and interviews involving individuals across the CKAN community spectrum, from stewards and maintainers to contributors and developers. Phase I's completion has laid essential groundwork for further advancement. Looking ahead, the application for Phase II sponsorship is underway. If successful, this next phase will focus on adapting and enhancing CKAN in line with the rapidly advancing technological environment.

Research: CKAN Instance Analysis

In an effort to map and understand the global distribution and characteristics of CKAN instances, the POSE Project has conducted a thorough investigation, resulting in the "CKAN Instance Analysis." This analysis provides a detailed overview of 381 functioning CKAN instances across 59 countries. These instances demonstrate a remarkable range in size, from modest collections to extensive repositories housing over a million datasets. Notably, they exhibit a dedication to consistent maintenance and updates post-launch, underscoring CKAN's broad adoption and its ability to manage diverse data volumes effectively. The research sheds light on the current landscape of CKAN deployments, showcasing the platform's remarkable scalability and flexibility. For a deeper understanding of these findings, the full analysis is available here: CKAN Instance Analysis.

CKAN Monthly Live

Throughout 2023, our CKAN Monthly Live sessions have been a vibrant platform for sharing, learning, and connecting with global open data leaders and innovators. Here's a look back at some of the highlights:

• Exploring Open Data Toronto
  • A notable session featured Mackenzie Nichols from Open Data Toronto who delved into the City of Toronto's Open Data Portal. The discussion centered around the portal's structure, the challenges faced, and the innovative solutions implemented. This session highlighted how CKAN facilitates accessible, high-quality data for citizens and beyond.
• Civic Engagement and Public Reforms with CivicDataLab
  • In another engaging meetup, CivicDataLab, an organization in India leveraging CKAN for societal betterment. They showcased their impactful work in projects like Open Budgets India, Open City, and JusticeHub, demonstrating CKAN's crucial role in public policy and civic engagement.
• Insights from U.S. National Science Foundation’s Pathways to Enable Open-Source Ecosystems (POSE) program
  • The POSE project team from the University of Pittsburgh and datHere shared their work on enhancing the CKAN ecosystem. They presented findings from ecosystem interviews, CKAN timeline, and map development, and insights from sense-making workshops, offering a comprehensive view of CKAN's growth and potential.
• Empowering Open Data in Africa with Emma Kisa
  • Our June meetup featured an enlightening presentation by Emma Kisa, Data Analyst and openAFRICA's Lead at Code for Africa. Emma delved into the emerging open data landscape in Africa, discussing the importance of strategic alliances and the role of CKAN in advancing data liberation.
• Warsaw's Open Data Initiatives
  • A session with Robert Kunicki focused on the Open Data Portal of the City of Warsaw. He discussed how the portal supports the city's transparency, citizen engagement, and smart city initiatives, showcasing CKAN's effectiveness in unifying and publishing city data.
• Swiss Open Data Community and Hackathons
  • Oleg Lavrovsky, a prominent member of the Swiss Open Data community, shared insights on engaging hacktivists and data explorers in open data initiatives. His talk highlighted CKAN's adaptability in supporting diverse events like hackathons and data quests.
• Advancing Data Management with Datapusher+
  • Joel Natividad introduced Datapusher+, a tool enhancing CKAN’s Datastore. This session highlighted its capabilities in data type inferencing and loading speed, underscoring CKAN's potential for analytical workloads.

Each of these meetups not only enriched our understanding of CKAN's applications but also strengthened our global community of data enthusiasts. We're grateful for the valuable insights and look forward to another year of inspiring discussions.

Launch of the CKAN Monthly Newsletter

A noteworthy initiative this year was the launch of the CKAN Monthly Newsletter. This regular update has become a key resource for keeping our community informed about ongoing developments, insights, and achievements within the CKAN ecosystem. Delve into past issues on our LinkedIn page to catch up on what you might have missed. And don't stop there – subscribe to the newsletter! Be part of the journey as we continue to explore new frontiers in open data. Your subscription is your ticket to staying informed and engaged with the ever-evolving world of CKAN!

Search Engine Update: Making CKAN Modular

CKAN is transforming the way we manage data by moving to a modular architecture, enabling the integration of various search engines beyond SOLR. This significant shift enhances adaptability, user experience, and technical flexibility, addressing the growing and diverse needs in the field of data management. For a deeper understanding of how CKAN is adapting to these changes and to explore the technical specifics behind this innovative approach, read our full article: Adapting CKAN to a New Era of Search Engines

Elevating CKAN's Interface

CKAN is enhancing its user interface to improve engagement and efficiency. This initiative underscores our commitment to continuous innovation in data management technology. We aim to make CKAN more intuitive and accessible. Stay updated on this transformative project here: https://github.com/ckan/ckan/issues/7760. Further details: A Call to UI Designers and Full Stack Developers: Help us Optimize User Engagement and Operational Efficiency for CKAN.

Product Market Fit Survey Insights

The release of the latest Product Market Fit survey offered interesting insights into CKAN's current position and potential growth areas. It revealed a positive reception among users, with a fit score of 64.3%. This survey, crucial for understanding user satisfaction post-updates and expansion, shows a high recommendation rate of 92.3% and highlights varied use cases from customization to active contribution and data publishing. It also identifies key beneficiaries of CKAN's features, including government entities, NGOs, and data hubs​. Read more here: Product Market Fit for CKAN (Autumn, 2023).

CKAN's Impact at Key Conferences in 2023

2023 saw CKAN's presence at several significant conferences, amplifying our voice in the open data community. Highlights included Ian Ward's insightful talk at the Canadian Open Data Summit (watch here) and Steven's engaging presentation on the architecture of trust in data ecosystems (view here). These sessions showcased CKAN's capabilities and fostered deeper connections with the global data community.

Linux Magazine Featured CKAN: Managing Open Data with CKAN

Linux Magazine has featured CKAN in an article titled "Managing Open Data with CKAN." This piece highlights CKAN's role as a versatile data management system that simplifies the process of building and managing open data portals. The article emphasizes the importance of ease in publishing, updating, locating, downloading, and integrating open data. It lauds CKAN for its robust set of tools designed to facilitate the creation of open data hubs and portals. Key features of CKAN, such as advanced geospatial capabilities, enhanced search functionalities, comprehensive metadata, and dynamic visualization options, are underscored as instrumental in managing and federating data efficiently. This coverage in Linux Magazine is a testament to CKAN's growing influence and utility in the open data community. Read the full article on Linux Magazine.

Conclusion

2023 has been a year of transformative growth and collaboration for CKAN. Each of these milestones represents a step forward in CKAN's journey to democratize data access and foster a more open, data-informed world.

As we celebrate these milestones, remember, our journey is far from over. We're only midway through our ambitious plans! If you're inspired to be part of this transformative journey, we warmly welcome you. There are myriad ways to engage with the CKAN Community. Discover how you can connect with us and contribute to the ever-evolving narrative of CKAN.

Join us in shaping the future of CKAN!