MultiRATE recently has developed the initial version of the Holistic RL assessment methodology
Deliverable D3.1 “MultiRATE Holistic RL assessment methodology-initial” that was delivered to the EC by the end of October 2023, presented the initial holistic assessment methodology. This report discusses the development of a robust and holistic assessment methodology. At the centre of the approach was the creation of a flexible and adaptable RL framework suitable for various scenarios. The effectiveness of the methodology will be tested, refined, and optimized through the upcoming project pilots, and feedback will be collected. Once the RL assessment is harmonized, it will facilitate an enhanced and coherent direction for system development assessment. Among its benefits is its ability to take the EU’s innovation diversity into consideration, acknowledging the multifaceted factors influencing technology adaption, and deliver a methodology that is both verifiable and scalable.
Drawing from the insights of D2.1 and an extensive literature review of existing maturity and readiness level frameworks, the Holistic Readiness Level (HRL) assessment offers a comprehensive perspective, going beyond a mere aggregation of discrete, domain-specific RL assessment methods. The ultimate aim is to develop a unity with the holistic RL assessment with due consideration given to domain-specific criteria.
The HRL assessment consists of two main phases: the initialisation phase and the actual assessment. The initialisation phase contains a number of optional steps, which are outlined in dotted lines in the figure. For the optional steps, the user has to decide which indicators they want to assess, whether they want to assign different weight factors to the indicators, and whether they want to set an ambition level per indicator. If they want to use the ambition level, they might also want to use a baseline HRL assessment to compare the current assessment with. This baseline HRL assessment is depicted by a single box in this figure, but it actually requires the user to perform both the initialisation phase and the assessment phase to determine the baseline.
The MultiRATE Holistic RL assessment methodology, as presented in this deliverable, relies on a comprehensive understanding of holistic readiness level assessment and the harmonisation between domain specific knowledge. The construction of this robust framework integrates and builds upon well-established scales and definitions, providing a foundation to enhance the trustworthiness of system evaluation processes. This, in turn, enables more straightforward acceptance and adoption of the development outcomes. The forthcoming stages in the project will refine and enhance its efficacy, promising a transformative tool for the realm of security.
MultiRate recently has published the D2.1 deliverable titled ‘’Cross-disciplinary RLs needs, methodologies and calculators-initial version’’.
The recently published Multirate deliverable D2.1 “Cross-disciplinary RLs needs, methodologies and calculators-initial version’’ initially presents the findings of a literature survey conducted on the mechanisms underlying technology maturity. It identifies the factors that contribute to maturity, describes innovation lifecycles, and highlights success and failure factors specific to EU-funded technology research projects. Additionally, it focuses on technology within the security sector and investigates whether the factors influencing technology maturity differ from those in other sectors. The deliverable further illustrates the cross-disciplinary needs for RLs, outlines user requirements for each RL, and presents the initial versions of the methodologies and indicators.
The deliverable D2.1 starts with describing a foundational knowledge base by providing detailed definitions of crucial terms and by introducing the MultiRATE conceptual model, a block-diagram of a generic readiness assessment methodology for a to be assessed element. The model describes a generic single readiness assessment such as TRL or CRL, IRL etc.
The MultiRATE conceptual model follows a three-step process. It begins with determining the element to be assessed by defining its scope, nature, and the environment in which it should be evaluated. Next, the criteria relevant to various stakeholder groups are identified, and finally, each RL is calculated within its operational environment using corresponding algorithms.
The deliverable also presents a comparative study along with an overview of differences in the market, stakeholders, financing related to innovation development and uptake among the research areas covered by the Horizon EU program, with a specific focus on the Security Research Domain. It discusses the characteristics and differences in market, stakeholders, financing, and innovation development and uptake within different clusters of EU research, such as Health, Digital, Industry and Space, Culture, Creativity, and Inclusive Societies, Climate, Energy & Mobility, and Food, Bioeconomy, Natural Resources, Agriculture & Environment, as well as the Security domain. These differences reflect the unique challenges and opportunities within each domain and highlight the need for tailored approaches to innovation development and uptake. The uptake of security innovations can be slow due to the need for regulatory compliance and the complexity of security challenges.
The 7 Areas of Readiness Level
In the past years, the European research community has identified that assessing readiness level is crucial as it enables a clear understanding of the technological maturity and feasibility of research outcomes. It facilitates informed decision-making, effective resource allocation, and timely identification of promising technologies for further development and implementation.
The ambition of MultiRATE is to develop a holistic, homogeneous, and harmonized Readiness Level (RL) evolution methodology and calculator for R&D projects and solutions in the security domain, which will be made available to the EU R&D community.
MultiRATE conducts research in multiple fields to create a solid ground for the development of the holistic, homogenous and harmonised RL evaluation methodology. It provides detailed information on each methodology, the indicators selected for each RL, and the respective calculator. Initially, existing methodologies were studied, assessed, and modified as needed for the development of each RL.
Subsequently, the MultiRATE maturity assessment frameworks were
created, proposing seven RL scales: Technology (TRL), Societal (SocRL),
Security (SecRL), Legal, Privacy and Ethics (LPERL), Integration (IRL),
Commercialisation (CRL) and Manufacturing (MRL).
A brief overview of these proposed RLs is provided below.
Technology Readiness Level (TRL) is defined as a mean for measuring or indicating the maturity of a particular technology during its acquisition phase. The TRL is used in each technology project evaluating the parameters and assigning a rate based on the project’s progress. The evaluation framework comprises four distinct indicator categories: Technology Preparation & Requirements, Documentation, Operability & Continuity, and Evaluation & Usability. These categories serve as key aspects for assessing the readiness and effectiveness of a system.
Societal Readiness Level (SocRL) is defined as a way of assessing the level of societal adaptation of, for instance, a particular social project, a technology, a product, a process, an intervention, or an innovation (whether social or technical) to be integrated into society. The evaluation framework comprises four indicator categories: stakeholder engagement, acceptance, social integration and social good.
The Security Readiness Level (SecRL) is a scale for measuring the security of an innovation or solution (a process or a technology) and its assets concerning its threat environment. The Security evaluation methodology, which will be developed within the scope of MultiRATE will consider security aspects beyond cyber, such as protection against human-caused and naturally caused incidents as well. Also, it covers measures for all different security functions, namely: Identify, Protect, Detect, Respond and Recover.
Legal, privacy, ethics Readiness Level (LPERL) tackles the maturity level of ethical risks, privacy, and data governance. Currently, no unified LPERLs exist for security or generalist technology or applications. To build the LPERL methodology it was chosen to rely on core European frameworks for ethical and privacy regulation or guidance. For each one of the aspects covered by the specific RL framework, dedicated indicators have been identified, such as legal indicators (e.g. Legal basis for data processing, Purpose limitation, Data quality and accuracy etc.) and privacy indicators (e.g. Lawfulness, fairness, and transparency, Integrity and confidentiality). By combining the identified frameworks and adapting them to the specific needs of security applications, a comprehensive and practical LEPRL evaluation methodology has been developed.
The Integration readiness level (IRL) defines a metric to measure the integration maturity between the technological components of a system. The assessment method that is utilised in the MultiRATE project will be based in an existing and validated IRL framework with the original decision criteria and indicators being modified and improved as needed to meet the demands of the project.
The Commercialisation Readiness Level (CRL) framework assesses various indicators which influence the commercial and market conditions beyond just the technology maturity. This enables key barriers to be addressed to support technology commercialisation. The evaluation framework comprises seven distinct indicator categories: Technological innovation, Market potential, Intellectual property (IP), Regulatory compliance, Business model and scalability, Team/Consortium expertise, and Funding and financial viability.
The Manufacturing readiness level (MRL) is a measure to assess the maturity of a given technology, component, or system from a manufacturing perspective. The evaluation framework comprises seven distinct indicator categories: Technical feasibility, Manufacturing process design, Supply chain readiness, Quality control and assurance, Manufacturing cost and scalability, Regulatory compliance, and Production capacity and timeline.
Funded by the European Union. Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or REA. Neither the European Union nor the granting authority can be held responsible for them.
Grant Agreement Number: 101073929
Funded Under: HORIZON-CL3-2021-SSRI-01
Funding Scheme: RIA – Research and Innovation Actions
Topic: HORIZON-CL3-2021-SSRI-01-01 – A maturity assessment framework for security technologies
Cordis link: https://cordis.europa.eu/project/id/101073929