AI is widely associated with large language models (LLMs), which are now increasingly used in people’s day-to-day lives, as well as workplaces. This technology is frequently harnessed for information retrieval, to synthesize complex topics, summarise long-form articles or reports, and process data. Many people also use AI for personal administrative tasks such as answering emails, setting up sales and marketing outreach and even technical troubleshooting. 

In the maritime sphere, LLMs are often tasked with analysing reports or regulations – particularly in the wake of meetings by the IMO and other regulators. A growing number of maritime workers are using AI to create reports for vessel operations, cargo or even port calls – with the added ability to translate these outputs into multiple languages. Charterers and brokers are using AI to process market and vessel data, companies are using AI to piece together market intelligence including fleet trends, and in some cases LLMs are even creating legal contracts.

While these solutions can prove valuable, we know that there are many questions about reliability. LLMs are known to ‘hallucinate’ and make up sources or pieces of information – and most include disclaimers with any outputs that note the importance of checking their outputs for accuracy. 

It’s with this in mind that we realised it’s time to put these platforms to the test, evaluating free-versions of the leading LLMs as part of our ‘De-mystifying digitalisation and AI in maritime’ campaign and assessing the extent to which they might deliver value for maritime businesses. 

Each month, we will pose questions about the maritime industry to the public versions of five leading AI platforms – ChatGPT, Gemini, Copilot, Claude, and Grok and invite an expert guest judge to help us assess which LLM made the grade. In some cases, the questions will be factual (with a definitive answer) while in others the answers being sought may be more open ended and creative/speculative.

We recognise this process is unlikely to be an exact science, but we have developed some key metrics that will allow us to evaluate these platforms as objectively as possible, before crowning a winner each month. 

Here’s how we’ll assess the free tier offering provided by the five leading platforms:

• Accuracy. What sources were used for the output? Is the information up to date and are factual errors avoided or minimised?
• Completeness. Do multiple perspectives ensure bias is avoided? Are all key facts included?
• Ease of understanding. Does the structure of information aid understanding? Is there a summary of the output that accurately captures the full picture? Are there follow-on questions suggested and are these relevant?
• Overall impression – What is the overall impression of the answer based on the personal judgement of the assessors. This allows for personal taste by the subject matter expert.

Over the next 12 months, we’ll be coming up with questions in partnership with our specialist judges – and we also welcome your suggestions. Is there a burning industry issue you’d like us to assess? Be sure to let us know – simply send us an email to suggest a question: info@intentcommunications.co.uk.