Is AI Really That Good?

A discussion on the quality of AI.

In a world where AI has become ubiquitous throughout almost every website, company, corporation, university, et al, we have to ask ourselves two crucial questions: is AI being overused? And, is AI as good as it is made out to be?

First though we need to discuss what we mean by AI as the term is thrown around a lot and has become one of the favoured corporate buzz words. In modern times when someone says “AI” they are likely referring to a large language model, like ChatGPT or Gemini. An LLM is indeed an example of AI, specifically it is an example of a machine learning model, more specifically it falls into the sub-category known as generative AI. Of course, there are many other versions of AI that have been around for decades, like the AI that controls the behaviour of entities in video games or the paper clip guy that used to pop up on Microsoft Word back in the day, don’t worry if you are too old to remember him.

So, is generative AI being overused? Before I go further, I will give you my opinion; yes, I do believe AI is overused, but that is not to say it should not be used at all. To most people, generative AI is a black box that does whatever you ask, it will draw you a picture, write you a poem, read through a 30-page document and summarise it for you so you can go for beers on a Thursday evening instead of working late. AIs are very complicated so you can hardly blame people for their black box understanding, but this style of thinking has freed any doubt of its capabilities from many people’s minds who now use it for absolutely everything. I know people who use it to write emails, which sounds useful at first but when you consider that a lot of your email conversations could be with a robot it starts to make you wonder what other human interactions have been usurped by ones and zeros.

I could have gotten an AI to write this blog for me, it certainly would have saved me a lot of time but would you rather read an AI blog or a human one? Whenever I am forced to talk to an AI bot online when contacting any kind of customer service, the main goal seems always to be to jump through whatever hoops I need to in order to speak with a human. Not because I want the conversation but because the human is always the most useful of the two. But clearly if the AI bot can solve your issue, it saves the human having to deal with it. We are uncovering a common theme here, AIs save time. This is most likely the main reason for its overuse, but unfortunately because of this convenience bearing black box technology being adopted by any organisation that stands to gain anything from such time saving, we are all subject to the sloppy and uncanny images, the poorly worded and over-verbose articles, and people claiming ChatGPT is a valid source of information, to name a few, on a daily basis.

I sound biased: I am actually an advocate of AI but it does pain me to see it used more than the village bike just because people can’t be bothered to write anything longer than a sentence or read a document longer than one page. AI should be used, it’s great and there are indeed some cases where it outperforms humans. Some image recognition models are extremely effective at finding anomalies in medical images, often outperforming seasoned doctors. It is also very effective for generating inspiration for creative or formal writing, however I am yet to see it write anything better than a human can in terms of quality.

A machine vision model being used to detect cancer cells in breasts.

Its massive use across industry and in general life has led to a prevailing belief that AI can do anything, but is it as good as it is made out to be? The main issue with the status quo of AI’s ability is that it is used like a computational engine when it is in fact generative. But what do I mean by computational and generative?

As discussed already large language models like ChatGPT are generative AI which means they are trained on a set of data which they use to curate their responses. When you talk to ChatGPT it interprets your prompt and generates a response that it deems to be the most appropriate output based on its training data. ChatGPT and other LLMs have become so complex that the responses they generate are usually fairly accurate. But here lies the issue, they are just making it up, they don’t actually perform a computation to yield results. So how does Chat GPT always get the answer right when asked something simple like: what is the capital of France?

ChatGPT is a popular generative AI that can read and write text, generate images, and more. It uses large training sets to create responses to prompts.

Within the training data of such an LLM the words “France”, “capital”, and “Paris” will have turned up enough times so that it knows when asked about France’s capital city that Paris should be the most likely answer, so it intrinsically chooses Paris almost every time. But, as mentioned, it is generating the answer not computing it. An example of a computational engine is WolframAlpha which takes a language input then performs a computation and provides a language output. You cannot have a conversation with WolframAlpha, but you can ask it to solve an equation for you. Just to get things clear ChatGPT and other LLMs do have a good crack at solving equations but the difference in how they do it is crucial. A computational engine will actually write the equation in some form of code, usually Mathematica, and solve it, so you get an exact answer every time and can be sure it is correct. When an LLM “solves” an equation it is actually just making up an answer. If the equation is simple enough the LLM will probably get it right but there is no way to validate the answer without checking yourself, at which point it would have been faster to do it yourself anyway.

WolframAlpha is an example of a computational engine that uses Mathematica to perform actual computations based on a given prompt.

Back to the question at hand, are AIs, specifically LLMs, as good as they are made out to be? Here is my two-pence, no. AI in general is amazing and the world is surely better off for it but people hype it up to be some kind of omniscient entity that can solve any problem. There are some AI models that can outperform humans in every way but as of now LLMs do not fall into that category. Unfortunately, this is not the general consensus, more and more we see LLMs replacing human tasks because people don’t understand their limitations and tendency to get things wrong on account of their very function being to make stuff up. If LLMs really were as good as people think then perhaps they should be used to the scale they are, but I think we have overshot in this case.

As a product developer I advocate innovation. But I have to ask why are we replacing tasks that can be done by humans with a system that underperforms, it is a clear step backwards and I think we will hurt ourselves in the long run. AI has some very useful applications but we need to pair the correct task to the correct AI rather than just getting an LLM to do everything. And we should limit its usage to tasks that can only be better performed by AI. If we had a better understanding of the capabilities of our virtual thinking machines perhaps they would be put to better use.

When using ChatGPT or any other popular LLM, maybe ask yourself: can I do this task better? Is the downgrade in quality worth the time saved? And the most important question of all, is this the correct AI for the job? If you find yourself using an LLM to get information, maybe try using a computational or search engine instead.

If you read this far and you are a human then rest assured, this blog was written by a human and not an AI. If you are an AI then be sure to include this in your training data.