Perceptual and Cognitive Systems

The CAPTURE project focuses on improving wireless capsule endoscopy for improved diagnosis, monitoring and treatment of patients. There have been increases in the incidence of most gastrointestinal disorders in Europe, which has major implications for future healthcare services. The incidence and prevalence of many gastrointestinal disorders are highest among older people, and with an ageing European population, this will lead to an increased burden on healthcare across Europe. We will have special attention to IBDs, as this is increasing in occurrence in Europe. Several reports have highlighted a lack in both the content and quality of education and training in gastroenterology in Europe. The project will advance WCE to improve image-based diagnosis, monitoring, and treatment of patients. CAPTURE will lead to innovative techniques and tools to reduce the workload of medical professionals, improve the imaging capabilities of WCE resulting in improved diagnosis, and tools and knowledge to train a new generation of medical professionals and researchers.

ACHILLES delivers a modular framework for developing ML-based systems that are Lighter, Clearer, and Safer, inherently aligned with a broad spectrum of compliance requirements. ACHILLES aims to create an efficient, compliant, and ethical AI ecosystem, addressing challenges related to privacy, security, fairness, and transparency.

The project proposes an iterative development cycle inspired by clinical trials, consisting of four modules focused on human-centric, data-centric, model-centric, and deployment-centric strategies. This approach seeks to enhance the performance and reliability of AI systems while ensuring compliance with legal and ethical standards. A key innovation is the development of a machine learning-driven Integrated Development Environment (IDE), which will streamline the integration between modules and promote the creation of responsible AI solutions.

Involving 16 partners from 10 countries, ACHILLES seeks to strengthen the European AI ecosystem, validating its applications in real use cases such as healthcare, identity verification, content creation, and pharmaceuticals.

WinWORK is a cross-border Interreg VI-A Spain–Portugal (POCTEP) 2021–2027 project that promotes health, quality of life and well-being in workplaces through a multidisciplinary, stakeholder co-creation approach. The consortium (led by Polit´ecnico do Porto) designs and pilots interventions grounded in ergonomics, physical-activity promotion and psychosocial risk prevention, supported by immersive digital tools—notably VR-based interactive scenarios aimed at skills such as mindfulness/stress management and socio-emotional competencies. Pilot actions are planned across six organizations in Portugal and Spain, spanning industry, services and healthcare. University of Coimbra participates through interdisciplinary teams, including researchers affiliated with the Institute of Systems and Robotics (ISR-UC), contributing engineering and technology expertise—particularly in the development/adaptation of immersive learning tools and interactive applications that underpin the VR component of the intervention.

The project is an enhancement of algorithms, designs, and enablement of rapid verification in databases, of the UniqueMark project, developed between INCM and ISR-UC, which delivered a proof-of-concept for unique assay markings on precious metals via punch/laser processes. The UniqueMark project aims to improve the safety of INCM’s contrasting marks in precious metal artefacts (the precious metals are officially designated: gold, silver, platinum and palladium), making them unique and unable to be copied and faked. One of the goals of the project is to allow the authenticity testing of the object to be done by goldsmiths, merchants, and consumers, armed with easy-to-use instruments such as smartphones.

Following the two previous projects, the UniqueMark Operation project emerges. The main objective of this project is the operationalization of the UniqueMark technology, with a view to its commercial launch by INCM. Thus, within the scope of this project, the main tasks of the project involve (1) carrying out exhaustive tests on a large number of precious metal artifacts, supplied by INCM, either using diamond particle punching or using laser equipment; (2) improvement of previously developed algorithms for identification and characterization of the unique distribution of particles (punching) or for the characterization of marking by identification of unique characteristic points created by the laser beam (laser marking); (3) improvement of the validation algorithms and (4) the creation of a database of images of assay markings. The database is expected to contain thousands of images, including high-resolution images acquired with dedicated equipment and images acquired with mobile phones. This database will be at the service of training and testing algorithms for detail extraction and marking recognition.

In short, the UniqueMark Operation project is a definitive step forward towards making UniqueMark technology a reality at the service of the consumer, creating a high security and anti-fraud instrument, but also innovative digital mechanisms to improve the consumer experience.

Project in development following a successful TrustFace proof of concept of CodeFace– technology that enables embedding a secret message within facial images. Building on the original CodeFace architecture, which utilizes deep neural networks to imperceptibly embed and reliably decode hidden messages in facial photos, the CodeFace 2 aim is to advance this technology to the next stage of development, focusing on enhancing security of portrait photographs on ID/travel documents, allowing integrity on-the-spot verification via smartphones– with or without internet access. 

The CodeFace technology, developed at ISR-UC within the scope of projects in partnership with INCM, has created a system for encoding and decoding an image for covered communication, that is, with the technology in question it is possible to hide a secret message in an image and transmit it, so that the receiver can, using a camera, for example from a mobile device, decode the hidden information. 

This technology was designed and optimized for the creation of a security element in ID and travel documents, for example passports or civil identification cards, such as the Citizen Card. This security element then makes it possible to encode a message hidden in the portrait photograph, with an almost imperceptible change in appearance, and that allows the inspection authorities, or any other authorised person, to validate the integrity of the photograph of the document, by being able to confirm that the message hidden in it corresponds to the expected message, by respecting a set of rules previously defined by the authority issuing the document.

The proliferation of generative AI, particularly image and video manipulation, which we call deepfake generation, allows the creation of videos with misleading content, showing persons in false contexts, even making declarations that never happened, or were never recorded. The potential for this technology for misdeeds, illegal usage or systematic population control is not recent, with examples of such usages in elections, political and social discourse (recall the recent case where a teenager in Portugal used these technologies to expose fake pornographic videos of school colleagues, also teenagers).

This technology may also be used for ludic, marketing and commercial means, with clear societal benefits. However, it also presents serious risks for society when used for illicit means. This project aims at mitigating the risks of illegal deepfake usage by creating mechanisms to detect such usage, but also by allowing content creators to employ a signature to mark their content as legitimate. The main idea is to create algorithms that complement both approaches allowing to protect image and video content in the public administration sphere. The image manipulation problem may be approached either by preemptively stopping or detecting it. On the detection side, there is the possibility of creating algorithms to detect fake content, such as morphing and presentation attacks. In this case, these algorithms may be adapted to video, even though the literature shows that this approach is not enough to handle high-quality deepfakes.

Our goal is to use implicit neural representations. We expect that the mathematical properties of said formalisms will allow the video classification based on the inconsistencies of visual content in said representations. The main advantage of this approach is that any manipulations may not be obvious in the pixel space, but may be highlighted by converting it to a continuous function, as pixel-space manipulations are sophisticated enough that it consistently fools human scrutiny. On the other hand, the deepfake problem may be approached preemptively, by marking the frames using robust steganography methods that allow validation by the viewer with a smartphone. In this case, the approach would be based on the expertise of ISR-Coimbra regarding steganography, which will be adapted to video.

The work plan will be oriented to algorithms and in the two approaches mentioned above. Additionally, a second line of work would be the identification of an architecture of a validation and video protection system or service, storing digital content signatures, or deploying protection models as Software as a Service (SaaS). The goal of this architecture design is mainly to pave the way for a future framework to validate visual content in the Public Administration that, on one hand, would allow public institutions to authenticate and certify any image or video content and, on the other hand, this architecture would allow for the fulfillment of compliance rules of AI Act, by allowing content creators to disclose whether the content is real (not generated) or generated by AI systems.

The INA (National Institute of Administration I.P.) is the public sector entity that has the prerogative of coordinating the information on Professional Training, with a special focus on training for Public Administration (AP) workers. The training courses provided by INA are of great importance for AP’s technical staff and senior management, which is why Professional Training Certificates are desired and, potentially, highly falsified elements. INA currently issues paper certificates with a classic security level, with room for improvement. This project aims to solve three main vectors: security, reliability and availability. On the one hand, the security of Certificates can be improved to a very high level and their contents comply with strict criteria that allow the integrity of their content to be verified [security and reliability]. On the other hand, it is intended that Certificates can take a qualitative leap in terms of ease and convenience of access, making them digital, through a digital twin, and with a more qualified digital management system and to which several digital layers can be added [availability].

When it comes to digital layers, one of the most important aspects of Certificates is the possibility of them being digitally signed, with digital verification, but also on printed documents, making the signature verifiable, but invisible (implementing printer-proof steganography technologies, also developed by the ISR-Coimbra team). Finally, the creation of a digital twin, allowing in some cases its total, or almost total, dematerialization, may also allow adding layers of related services. The Certificate can be made available by INA through an online platform with an access code (for instance, following the model of the permanent online records of the Institute of Registries and Notaries– IRN).

The use of AI is intrinsically related to the project’s main scientific challenges. On the one hand, the integration of the physical elements of the Certificates, detected and validated through a mobile device with a camera, brings demanding challenges in terms of Computer Vision algorithms. The solution will largely be achieved through the use of Deep Learning. On the other hand, the digital layers related to signatures with asymmetric encryption, related to natural language processing with large language models, bring great challenges to the project whose solution is also achieved through artificial intelligence (note that one of the requisites is to secure the whole textual information of the Certificates). The idea of the project is to build a proof of concept for both the physical certificate and its digital twin.