Since early 2023 Marla Blue’s engineers have been working hard on our bespoke model to predict the underwater visibility for divers around the UK and Ireland.
We’ve done hundreds of experiments, used years and years of coastal data and 100s of diver reports to tweak, refine and improve our model.
We’ve managed to create a model that produces a visibility forecast, last week releasing version 3! So how do we know how accurate the numbers are?
Over the past few months we’ve been asking divers to report the visibility from their dives. We need 100s of visibility reports to analyse and compare the real in-the-water visibility to our models forecasts.
We’re stoked to say that we gathered over 500 visibility reports across the UK this diving season 🎉.
58% of the visibility reports we received were estimated within Marla’s (new V3.1) predicted ranges
This number might not tell the full story, and there is a lot more we can unpack. In this blog post we will walk you through a detailed analysis of the data you provided and share more about accuracy metrics for all iterations we did on Marla’s algorithms.
Let's dive right into it (pun intended).
Analysis of visibility reports submitted to Marla
We’ve had almost 500 reports submitted to Marla until October 31st 2024; the majority of them after our call to action on July 24’.
We received reports from all across the UK and Ireland, most of them coming from the South-West (287) and North-East (121) regions.
The most popular dive spot was Beadnell point with 22 reports (kudos to Scuba Steve). We’ve even had a couple of reports from lakes and quarries!
What visibility was recorded? Overall, most of the good visibility was recorded in the South-West and on the coast of Ireland. The highest visibility reported was near Saint Abbs on 15th of May 24’ with more than 20m.
Average visibility reported was 4.12m with a median of 3.5m and variability (standard deviation) of 2.86m.
Marla blue performance compared to diver visibility reports
How did Marla’s model compare to the diver reports? We’ll focus on the performance you’ve seen so far in v2/v1 models, and how a newly trained v3.1 model has improved.
To evaluate performance, we’ll report a single metric which calculates the % of times submitted report was within Marla’s range.
Our machine learning system consists of a forecaster model and a calibrator model. In v2 models, we’ve improved the forecaster model, by training it on a large data set of around 6 years of satellite observations. Differences between v2.1, v2.2 and v2.3 lie in exact variables (like offshore/onshore wind, chlorophyll concentration, waves) and time-spans chosen for training.
In v3.1, we’ve improved the calibrator model, by re-training the model with all of your diver reports.
In the background, we have a live monitoring system recalculating KPIs (key performance indicators) for each viz report that is submitted to Marla. The numbers below are calculated for a sample of n = 142 recent reports, albeit v3.1 number is interpolated from an experiment run on other 85 samples to avoid mixing datasets between training and testing.
The data shows that 58.01% of the diver reports were within the range forecasted by v3.1. This is a 28.8% improvement over v2.2 (our most recent live model).
Limitations and next directions
To train Marla, we used two different sets of visibility reports. One set came from various external sources, and the other set is reports submitted directly to Marla.
We’ve found that Marla’s users tend to report more often when prediction is incorrect, which leads to smaller overall visibility numbers compared to other sources.
This is likely the reason adding more reports from our website had such a large effect on algorithm performance.
In future, we are planning to incorporate diver reports more directly into our real-time forecasts. We believe this will make our algorithms more agile and reactive to user feedback, perhaps also enhancing our confidence score.
We need your feedback
Let us know whether you notice the difference after this recent release. We’re always curious to learn from fellow divers, reach out to us at info@marla.blue.
We’ve still got work to do, and we need your help by reporting the visibility whether Marla is right or not. We’re confident that we’ll be able to increase the model accuracy and provide you with better underwater visibility forecasts that will make your dive planning much simpler.
We love data. If you keep a digital log book of your dives (we need the GPS or lat/ log, dive date, visibility) and would like to contribute to research, email your log books to info@marla.blue
Comments