From black boxes to explainable AI

My first months at NimbleNova.

After nearly a decade in the tech industry, I thought I had a solid understanding of this sector. I’d helped clients solve complex problems and managed accounts across cutting-edge projects. But diving into the world of AI has been a game-changer. When I joined NimbleNova (a boutique advisory firm dedicated to solving real-world challenges by making organisations mature in their data and AI usage), I quickly realised how transformative and, at times, mystifying AI can be.

Our mantra at NimbleNova is “working together… with data,” and it perfectly captures the collaborative spirit of making AI work for real-world applications. Even with my tech background, concepts like Large Language Models (LLMs) proved to be a fascinating learning curve. At the same time, I saw firsthand where organisations struggle with AI’s limitations, from its complexities to the ever-present "black box" problem.

Surprisingly, even in the tech world, AI can feel mysterious to many. That’s why I’m writing this blog, for people like me, who have a foundation in technology or an interest in learning but still find themselves asking: What is AI really doing behind the scenes?

If that resonates with you, keep reading. We’ll explore the basics of LLMs, unpack their capabilities, and examine why they’re reshaping the way we think about technology.

LLMs made simple: the basics behind modern AI

To understand LLMs (Large Language Models), it helps to briefly explore their origins. AI has evolved through different layers of technologies, each contributing to the capabilities we see today.

• Artificial Intelligence (AI): The broad concept of machines mimicking human intelligence, from recognizing speech to making predictions.
• Machine Learning (ML): A subset of AI focused on systems that learn from data to improve over time without explicit programming.
• Deep Learning (DL): A type of ML that uses neural networks to solve complex problems like image recognition and language translation.
• Natural Language Processing (NLP): The branch of AI that deals with understanding and generating human language, making chatbots and translation tools possible.

LLMs represent the culmination of these advancements. They are specialized AI systems built using deep learning and NLP to process, understand, and generate human-like language. Trained on massive amounts of text data, LLM applications like ChatGPT can answer questions, summarize information, create content, and much more.

Key areas of LLMs

LLMs are not all the same. They vary in their design, capabilities, and use cases. Here are 3 key areas to help you understand the range of their applications and challenges:

1. Generative AI: a creative playground

Generative AI allows LLMs to go beyond understanding language to create it; producing new content like stories, poetry, or even code. Applications like ChatGPT shine in creative tasks where reliability isn’t critical. For example, brainstorming ideas or crafting fictional narratives.

The errors are obvious:
Errors, like nonsensical text or factual inaccuracies, are often easy to spot and harmless in these settings.

However, this creativity comes with a trade-off: hallucinations. Hallucinations are responses that are plausible but factually incorrect. While tolerable in art or brainstorming, these inaccuracies make generative AI less ideal for precision-focused tasks, such as medical diagnoses or financial planning.

2. Black Box AI: where trust falls short

Many generative AI systems are “black boxes,” meaning their internal decision-making is hidden. While this is acceptable for creative tasks, it becomes a major concern in high-stakes fields like healthcare, government, or finance. Without understanding how AI generates its outputs, users struggle to trust its fairness, accuracy, or reasoning.

For example, an AI recommending a medical treatment or assessing loan eligibility could rely on flawed or biased data. Worse, it might produce hallucinations (plausible-sounding but incorrect responses) that go unnoticed in critical contexts.

In these scenarios, transparency is essential. Users need to know not just what the AI decided, but why. This demand for trust and accountability has fuelled the rise of Explainable AI, which aims to make AI processes clearer and more reliable for high-impact applications.

3. Explainable AI: building trust in LLMs

Explainable AI (XAI) is an approach designed to shed light on these “black box” systems. By making AI decisions transparent and traceable, explainable AI fosters trust in high-stakes environments like healthcare and finance. For example:

• Retrieval-Augmented Generation (RAG): combines LLMs with external databases to improve accuracy.
• Knowledge Graphs (KGs): provide structured, traceable data relationships, reducing the chance of misinformation.
• Vectorized Models: enable efficient data searching and comparisons, helping AI explain how it arrived at specific results.
• Augmented Intelligence (AIX): enhances human oversight, blending AI’s efficiency with human intuition.

Bridging the gap: from creativity to reliability

Generative AI thrives in creative contexts, offering endless possibilities for content creation, brainstorming, and artistic exploration. But when accuracy, trust, and accountability matter, explainable AI steps in. By combining the creativity of generative AI with the transparency of explainable AI, we’re moving toward systems that are not only innovative but also dependable.

For example:

• In the government sector, explainable AI can analyse appeals by providing transparent, evidence-based reasoning to support decisions, fostering trust and accountability.
• In the procurement domain, it can clarify supplier contract obligations, match them with delivered services, and identify discrepancies, ultimately driving better value for money and improved oversight.

This blend of creativity and reliability ensures AI can adapt to a wide range of use cases; paving the way for broader adoption across industries

What’s next?

The world of LLMs is rapidly evolving, with new breakthroughs bridging the gap between innovation and trust. In future posts, we’ll delve deeper into:

• How LLMs work: a closer look at Retrieval-Augmented Generation (RAG) and vectorized models.
• Knowledge Graphs: how structured data is revolutionizing AI transparency and intelligence.
• Real-world use cases: exploring where and how LLMs are making the biggest impact today.

Sneak Peek!

At NimbleNova, we don’t just guide organisations on their AI journey, we stay ahead by conducting cutting-edge applied research in Explainable AI (XAI). This hands-on approach ensures that the advice we provide isn’t just theoretical but rooted in real-world applications.

Curious to see XAI in action? Let me show you how we use it in our daily work. Just reach out to me, or contact us at NimbleNova, and I’d be happy to share more!

Share this blog: