Cybersecurity protects our information. Firewalls, encryption, identity management, and security monitoring form the foundation of nearly every digital organization today. But with the advent of artificial intelligence, a fundamental assumption is changing: Perhaps in the future, we will no longer need to protect just information but decisions as well. What does this mean for companies?

A Look Back and Ahead

Artificial intelligence is fundamentally transforming businesses and the world. Large language models, autonomous agents, and intelligent assistance systems automate processes, support decision-making, and create entirely new business models. At the same time, however, risks are emerging that go far beyond traditional information security. Prompt injection, data poisoning, manipulated models, compromised AI supply chains, hallucinations, uncontrolled agents, and violations of regulatory requirements clearly demonstrate that AI security is no longer merely an extension of cybersecurity it is a discipline in its own right.

IAIQS as a vision for trustworthy AI. Today, international standards such as ISO/IEC 42001, the NIST AI Risk Management Framework (AI RMF), the EU AI Act, MITRE ATLAS, and the OWASP Top 10 for LLM Applications address precisely these issues. Yet many of the fundamental principles had been recognized by scip AG years earlier.

As early as late 2018, scip AG developed IAIQS (Interdisciplinary Artificial Intelligence Quotient), a framework dedicated to the quality, safety, and trustworthiness of artificial intelligence.

At that time, chatbots and generative AI were not the focus of public discussion. Instead, scip AG explored fundamental questions:

• How can trust in AI be made measurable?
• How can quality and safety be assessed together?
• What risks arise from learning systems?
• How can humans remain accountable at all times despite automation?

These questions form the core of modern AI governance today and are reflected in nearly all current international AI frameworks.

Humans Remain the Most Important Safety Component

Since the release of ChatGPT, artificial intelligence has been experiencing unprecedented momentum. Companies are integrating large language models into business processes, autonomous AI agents are taking on complex tasks, and intelligent systems are increasingly making decisions that, just a few years ago, were reserved exclusively for humans. With this development, the question itself is shifting.

The decisive challenge is no longer “What can AI do?” but rather “When can we trust AI?”

After all, an AI solution is not automatically trustworthy simply because it delivers impressive results. Trust is built through security, transparency, traceability, quality, and responsible governance. This conviction has guided scip AG for many years. While many organizations focus on the performance of their models, we have always taken a different approach:

The focus is not on AI, but on people.

AI is meant to support people, make decisions more transparent, and make processes safer. However, it must never completely assume responsibility for critical business decisions. This human-centric approach aligns with the fundamental principles of the EU AI Act as well as international AI governance models, which define transparency, traceability, human oversight, and accountability as key requirements.

AI security begins long before the prompt

When people talk about AI security today, they often think first of the language model itself. The focus is on topics such as prompt injection, hallucinations, or the misuse of chatbots. However, this perspective falls short. In fact, the security of an AI system does not begin with the first user input, but rather long before that, during the development of the entire system.

A modern AI system is not a single application, but a complex ecosystem of numerous interconnected components. Data is sourced from a wide variety of sources, processed, and stored. Models are trained, fine-tuned, and continuously updated. APIs connect external services, vector databases provide knowledge, identity services manage permissions, agents orchestrate tasks, and cloud infrastructures ensure operation. This interplay is complemented by software supply chains, monitoring, logging, and numerous other technical building blocks.

Each of these components expands functionality, but at the same time also increases the attack surface. The security of an AI system is therefore not determined by the model alone, but by the security of the entire architecture. This is precisely where AI security differs from the traditional approach of focusing on a single model or chatbot.

• A Retrieval-Augmented Generation (RAG) system provides a clear example. The language model can function completely unchanged and technically correctly. However, if the underlying knowledge base is manipulated or if specifically crafted documents find their way into the vector database, the model will still provide incorrect or misleading answers. From the system’s perspective, the AI is functioning correctly; it is simply processing compromised information. The actual target of the attack was therefore not the model, but the database.

• The situation is similar with AI agents. They are increasingly being granted far-reaching permissions to independently send emails, query databases, manage cloud resources, or execute business processes. If such an agent is compromised or tricked into performing unwanted actions through clever manipulation, the resulting risk extends far beyond the quality of a single response. AI is evolving from an information system into an active agent and thus into a security-critical component of the corporate infrastructure.

• Training data also presents an attractive target for attacks. Even small, targeted manipulations can have a lasting impact on a model’s behavior. Such data poisoning attacks alter the AI’s knowledge base and are often difficult to detect later on. Equally critical are compromised open-source components, insecure model weights, or manipulated libraries within the AI supply chain. Since modern AI platforms rely on a multitude of external frameworks and dependencies, the security of the entire software supply chain is becoming increasingly important.

At the same time, new attack techniques specifically tailored to AI systems are emerging. Prompt injection manipulates the behavior of a language model through cleverly crafted inputs. RAG poisoning influences external knowledge sources so that false information appears trustworthy. Model manipulation targets the behavior or integrity of the model itself, while compromised agents can carry out autonomous actions against the interests of their operators. Together, these attacks demonstrate that the focus of cybersecurity is increasingly shifting from traditional IT systems to complex, learning, and partially autonomous systems.

This development is now also systematically described by international security frameworks. The MITRE ATLAS framework documents the tactics, techniques, and procedures attackers use to compromise modern AI systems. In addition, the OWASP Top 10 for LLM Applications describes the most significant current risks associated with the use of large language models ranging from prompt injection and insecure plugin architectures to inadequate access control and data leaks.

The key insight is therefore as simple as it is far-reaching: AI security does not start with the prompt. It starts with the architecture, the data, the processes, and the people who develop and operate an AI system. Those who focus exclusively on protecting the model overlook a large portion of the actual attack surface. Only a holistic view of all components lays the foundation for trustworthy, resilient, and long-term secure artificial intelligence.

Holistic AI Security Combines Governance and Technology

The greatest challenge today is combining technical security with governance. This is precisely where the added value of a holistic consulting philosophy comes into play. The discussion surrounding AI security often focuses on individual forms of attack, such as prompt injection or jailbreaking of large language models. While these threats are undoubtedly relevant, they represent only a small fraction of the actual attack surface of modern AI systems. Today, a productive AI system consists of a multitude of interconnected components. In addition to the actual language model, these include data sources, vector databases, Retrieval-Augmented Generation (RAG) components, API gateways, identity services, cloud platforms, agents, external tools, and, increasingly, Model Context Protocol (MCP) servers and autonomous AI agents. Each of these components has its own security requirements and can become a target for attack.

• A simple example illustrates this complexity: A company develops an internal AI assistant that can access project documentation, technical specifications, and customer data. The language model itself operates flawlessly and is well protected against classic prompt injection attacks. Nevertheless, an attacker could use a misconfigured vector database or inadequately secured permissions to retrieve confidential documents that fall outside the model’s scope of responsibility. In this case, the actual security problem lies not in the model, but in the architecture and the connected systems.

With the increasing prevalence of autonomous AI agents, the threat landscape is shifting further. AI agents no longer interact exclusively with users but perform actions independently: they send emails, manage appointments, access ERP or CRM systems, write code, configure cloud resources, or orchestrate complex business processes. A compromised agent may therefore have significantly broader permissions than a traditional user. Accordingly, identity management, fine-grained authorization, runtime monitoring, and continuous logging are becoming considerably more important.

The quality of the underlying data is also a key component of security. Faulty or manipulated training data can lead to systematically incorrect decisions. If the knowledge databases of an RAG architecture are compromised, the model may provide technically correct but substantively manipulated answers. This form of manipulation is particularly critical because it is often virtually undetectable to users. The integrity of training data, embeddings, and knowledge sources thus becomes a central security requirement.

At the same time, technical security alone is not enough. Companies must define which AI applications may be used at all, what data may be processed, and who assumes responsibility for development, operation, and continuous monitoring. Without clear governance, uncontrolled shadow solutions quickly emerge, in which employees use public AI services and thereby unintentionally disclose confidential information or violate regulatory requirements.

This is precisely why international frameworks such as ISO/IEC 42001, the NIST AI Risk Management Framework (AI RMF), the EU AI Act, MITRE ATLAS, and the OWASP Top 10 for LLM Applications are becoming increasingly important. They are all based on a common principle: Trustworthy Artificial Intelligence is not created solely by secure models, but through the interplay of governance, information security, data protection, software development, identity management, continuous monitoring, and human oversight.

scip AG has been pursuing precisely this approach since the development of the IAIQS framework in 2018. Even back then, the focus was not on the performance of individual models, but on the question of how to systematically build and maintain trust in AI systems over the long term. This idea continues to shape today’s consulting philosophy: AI security does not begin with the prompt and does not end with the model. It encompasses the entire lifecycle of an AI solution, from strategic planning through secure architecture and development to continuous improvement. Only when governance, technology, and people are viewed as an interconnected system does an AI landscape emerge that is powerful, secure, and trustworthy in the long term. This is where AI Assurance by scip comes into play.

Cybersecurity protects information | Trustworthy AI defines principles | AI Assurance builds trust in artificial intelligence

A secure AI landscape encompasses AI governance, information security, data protection, cloud security, identity and access management, secure software development, data governance, model lifecycle management, security monitoring, incident response, business continuity, human oversight, and more. These areas must not be viewed in isolation.

Research as a Driver of Innovation

Technology is evolving faster than ever before. New frameworks emerge within a few months, attack techniques spread in a matter of days, and artificial intelligence is changing the way we work almost on a weekly basis. Anyone who, in this environment, relies exclusively on established standards, certifications, or best practices will inevitably fall behind reality. Standards describe what was known yesterday. Security, however, must be geared toward the challenges of tomorrow.

For this reason, research has been one of the company’s cornerstones since scip AG was founded. It is neither a separate innovation lab nor a marketing tool, but rather an integral part of our daily work. Every insight gained from a penetration test, every red team operation, every incident response investigation, or every security analysis can serve as a starting point for new research questions. At the same time, new insights from research find their way back into client projects. Research and practice thus do not form a linear process, but rather a continuous cycle of knowledge.

This self-image has shaped scip AG for more than two decades. Long before generative AI became the dominant technology topic worldwide, the research teams were already examining the security implications of intelligent systems. In 2018, the IAIQS was launched as a research project that deliberately examined artificial intelligence from more than just a technical perspective. The goal was to understand security, governance, ethics, society, and technology as an interconnected system an idea that is now receiving increasing international attention.

The motivation was never simply to jump on a trend as early as possible. Rather, the focus was on the conviction that new technologies must first be understood before they can be secured responsibly. It is precisely this understanding that continues to shape scip AG’s research work to this day. The results of this work have been publicly shared for many years through scip Labs. The Labs serve as a platform for open knowledge transfer. Instead of marketing content, they produce technical analyses, security studies, proof-of-concepts, research findings, and practical insights drawn from real-world projects. Many of these publications address topics long before they appear in regulatory requirements or industry standards. This early identification of technological developments makes it possible to analyze risks before they have already become commonplace.

The thematic breadth of the research reflects the complexity of modern IT landscapes. Traditional disciplines such as penetration testing, Active Directory security, cloud security, mobile security, industrial security, and red teaming continue to form the foundation. At the same time, new fields of research are coming to the forefront: large language models, agentic AI, AI supply chains, prompt injection, model poisoning, adversarial machine learning, secure AI architectures, AI governance, and detection engineering. It is precisely at the intersections of these disciplines that the crucial questions for trustworthy artificial intelligence are emerging today.

Another long-standing focus has been on international vulnerability research. Security vulnerabilities arise daily, are constantly evolving, and often serve as the starting point for complex attack chains. As a founding member and active supporter of VulDB, one of the world’s most comprehensive vulnerability intelligence platforms, we have unique insight into the global threat landscape. The insights gained there are directly incorporated into security assessments, attack simulations, and strategic consulting engagements. This fosters an understanding of risks that goes far beyond the analysis of individual vulnerabilities.

The same applies to offensive security research. In numerous red-team projects for international companies, modern attack methods are not only analyzed theoretically but also replicated and refined under realistic conditions. This practical knowledge forms an indispensable foundation for realistically assessing new attack techniques against AI systems, autonomous agents, or complex enterprise platforms. Many risks can only be understood once one grasps how they are actually exploited in practice. However, research does not end with the analysis of existing technologies. It also encompasses the development of proprietary tools, assessment methods, and platforms. Whether it involves automated testing frameworks, vulnerability intelligence, local AI systems, or new evaluation models for artificial intelligence, the goal is always not only to observe technological developments but also to actively shape them.

Particularly in the field of AI assurance, this integration of research and practice becomes a decisive factor for success. The pace of technological development now far exceeds that of traditional security processes. Anyone who wants to develop or evaluate secure AI systems today cannot, therefore, rely exclusively on existing standards. They need a deep technical understanding, ongoing research, and the willingness to continually question assumptions critically.

Ultimately, trust is not built by consistently applying known solutions. Trust is built where new risks are identified early on, scientifically investigated, and translated into practical safety measures. Research is therefore not an end in itself. It is the foundation for ensuring that innovation can be used responsibly and is thus one of the most important elements of AI assurance.

Conclusion

The future belongs to trustworthy AI. Trust is never one-dimensional. The real challenge is not to deploy artificial intelligence as quickly as possible. The challenge is to deploy AI in a way that is safe, transparent, and sustainable.

Trust is the true currency of artificial intelligence.

We have been conducting scientific research on this topic since 2018 (1, 2, 3, 4, 5) with trust in AI. As early as 2018, the IAIQS approach laid out a vision that is now validated by international standards, regulatory requirements, and technical frameworks: Trustworthy AI does not arise solely from powerful models. It emerges from the interplay of research, cybersecurity, governance, quality management, and a consistently human-centered approach.

We don’t just secure AI systems. We build trust in AI. From the initial idea through development to continuous improvement. This trust is based on five principles:

• Research-driven: Insights from the scip Labs are directly incorporated into methods and assessments.
• Security by Design: Security is integrated into AI solutions from the very beginning.
• Human-centric: Humans remain responsible for decisions and governance.
• Lifecycle-oriented: Support throughout the entire AI lifecycle, from strategy to further development.
• Independent: Vendor-neutral consulting with an objective assessment of risks and opportunities.

Anyone who wants to use AI successfully in the long term therefore needs more than just modern models. They need a security strategy that takes technology, organization, and people into account equally one that is holistic, sustainable, and scientifically sound.

We build trust in artificial intelligence.

A new interdisciplinary field will gain importance in the coming years. A field that deals not only with information, but also with decisions. We call it AI Assurance by scip. Following the successful completion of our first client projects, we now will be presenting our approach as part of a research initiative over the coming months. You can find a first glimpse here.

About the Author

Simon Zumstein has been working in IT since the 1990s as an engineer, project lead, security consultant and CIO. Integral risk management while taking managerial-economic factors and the presentability to decision makers are his area of expertise.

Links

• https://artificialintelligenceact.eu/
• https://atlas.mitre.org/
• https://owasp.org/www-project-top-10-for-large-language-model-applications/
• https://www.iso.org/standard/42001
• https://www.nist.gov/itl/ai-risk-management-framework
• https://www.scip.ch/?ai_assurance
• https://www.scip.ch/?labs.20180823
• https://www.scip.ch/?labs.20181018
• https://www.scip.ch/?labs.20190718
• https://www.scip.ch/?labs.20191205
• https://www.scip.ch/?labs.20250710
• https://www.scip.ch/?labs.20260212
• https://www.scip.ch/?labs.archive
• https://www.vuldb.com