Jev can finally see. Last week we did a benchmarking tool against Jev, Clev, Nimble, and Tev. However, Jev was completely left out because it does not have image capability. Now they have released it. We are going to run it against Clev and Clev Flash, which have already had image capability, but also the Perplexity decider model and GPT6 Luna, the OpenAI decisions API. We are going to be running each of these models through 3 sets of vision based tasks with increased difficulty. Clev, Clev Flash, and Perplexity are all going to be run on device. There is an open local model solution there for comparison against these more remote models. I do not know what more else I can say. Let us just get on with the benchmark. So, I have built this little benchmarking tool and we are going to go with this data set here. Let us just start the run and see how we get on. It is 150 photos. So, we are going to be seeing a delay. But what this is doing is running the images against a known result of what they should actually answer to. We have decided what the answer is and we are seeing if the decision model can match that. So, let us have a look at who is holding the man's feet. I guess stirrups. Yeah, stirrups. I got that right. Hey. And Jev got that right. GPT6 Luna got that right. Clev was 97% sure it was stirrups. Why is the man holding the fork? Well, I guess because he is eating and it is necessary for cutting food. I would not necessarily say that, but out of the options, I guess Jev is 99% and Clev is 98%. So, that is kind of what is going through here. And we can see the results here. But, let us just sit tight, wait for the benchmarks to run, and we will see the result in a second. Okay, the results are in. So basically the leader is Jev with 92% and here is the leaderboard. But it is worth looking at some of these other figures. P50 is the fastest response time on average. Jev is obviously being very fast here. OpenAI decisions API is actually very fast. That is the fastest one. Here we go. We can see the fastest P50 there. However, the P95 is what the slowest result is like. What were you likely to experience at slow times? Really, I mean you are looking at Perplexity here. Both numbers are over 2. I am not really sure why you would ever go for this sort of model. Clev Flash is consistently being under 1. It lives by its name, I guess. However, the accuracy is likely to take a fault. The calibration: this one here is basically how confident is it against right answers. If it is really confident but it is the wrong answer, then that gets deducted 2. If it is hedging its bet 50/50, it is adding 0.5. Basically, we are looking as close to 0 as possible. So the winner here is Clev Flash being just -1. Now these are the costs. So again, are we really bothered about pricing of these models at this stage because it is so freaking cheap? We ran 150 tests against that and it costs less than $0.01. Now I will let you check the screen now on categorisation. This might be useful to you on understanding which categories it shines in. But let us take a look at some of these. So this one here: which of these people is very warm? Now, that is a very interesting question because the woman on the left there, I guess they are answering the answer is both. I guess because they are using jumpers or something. Both people are very warm. Interesting one. I mean, even as a human, I am struggling with that one. So, Jev confidently said see the left person, which is what I probably would have said because they are kind of closer to the candles. Not sure how this person would necessarily be warm. If anything, they might be cold because they both have jumpers on. No idea. Look at this one here. How many skaters have both feet on the board? Well, only 1 of them because this one is doing like a... well, I do not even know. Can you see that there? I suppose he has his foot on the board. Both of them. There are 2 skaters. How many skaters with both feet on the board? 2. That was that. Jev got that one right. Luna sucked at that one. And Clev got that right. Clev Flash did not get that right. Or Perplexity decider also did not get that right. Overall, I am just going to say it, but I am very disappointed. Perplexity decider: that is a 27b model and it is slow and it is not getting things right. So, interesting one. I will post these results so you can look more in detail. But yeah, I guess if anything matters to you, as always, it is probably worth just checking against your own data sets to figure out what your use case is and how good these models are depending on your use case. So there you go. If you wanted a little rundown on how these models perform, you can go home and carry on with your day. However, I have done the Blink Hard Perception Test here, which humans get 95% right. Vision models do particularly badly at this data set. Get it on Hugging Face. It is things like: which image shares the same style as the reference images? The reference images have not loaded but you can see of course that would be a difficult question to answer. I guess they are all about the image styles here. It is less of a text based answer, more of an image based answer based on an image. I have also created an interesting test where the option is that you cannot tell from the image what is happening. And that is interesting because it shows where the models place confidence where there is not a lot of confidence in the question. It is a very ambiguous answer to the question. So, it should say it cannot tell. It should hold its hands up. So, let us run the Blink Hard Perception Test and see how they get on. Okay, so here we go. Jev coming out on top there with an accuracy of 71% and pretty fast to be honest, but I am keeping my eye on this calibration. So, again, this is where it is saying it is pretty confident against a wrong answer. We want to get as close to 0 as possible. Cost is the same really. It is negligible. Clev coming out 2nd there, but actually being quite slow as well as uncalibrated. I guess it just got lucky there in those instances. Decider coming out just a little bit underneath and still quite slow but pretty well calibrated. Flash and surprisingly GPT6 Luna or the decisions API getting 62% accuracy. It is still pretty fast and very comparable to Jev, but just not getting the right answers. So, we can see here: how many words on the sign in this image have the letter I in them? Jesus Christ, I can barely see that. So, what is that? 1, 2, 3. Answer being 3. Clev being the only one that gets that right. So, yeah. And here is the pictures. And I guess that is like humanity's last exam. This one. Jesus. Luna getting that one there. Let us have a look. That one is a pretty easy one. It does look like we got some errors there on that result. So something to bear in mind. See there. Yeah. So, we are getting errors for Jev, unfortunately, on a lot of these images, which is probably... well, I mean, it still came out on top, but it probably lost a lot of points during these ones if the API is sending through the wrong size image. And this is where the models have been given the option that it cannot tell, and it is whether it chooses that option because a lot of these you actually cannot tell. GPT6 Luna and Perplexity coming out on top on this one. Same story about speed really. I mean, the decisions API is a fast model and Perplexity is a slow model. So, not a lot of surprises there. Jev coming out 92%. They are all pretty close to be honest. Not as dramatic as the previous one dropping to the low 60s. We come and look at this one. Let us take... okay, this woman just heard some life changing news. So, very, very ambiguous. She is just looking at the Eiffel Tower there. This woman just heard some life changing news. What is her response most likely to be? Cannot determine, of course. Rage, destruction, silence. So, yeah. Cannot determine. Luna getting that one right. And Perplexity. Yeah. Who will win the game that they are playing? Well, there we go. Cannot tell. Of course, it is cannot tell. All of them getting that right. What will this girl be when she grows up? Cannot tell. What did Clev say? It thought she was going to be a chef. I mean, to be fair, she is cooking. She is cooking. There is definitely an interpretation there to be had. So, yeah, some really interesting results on that one. I will post all of the results on there so you can dig in and have a little look. There will be a link down below in the description. Overall, I am seeing a really interesting... especially with these vague tests or these sort of... I know it is very insightful when you give someone a photo of someone sat in front of the Eiffel Tower what they are thinking and feeling. I think there is a really insightful way to determine the AI and its kind of undercurrent of biasness or whatever. I do not know. It is interesting nonetheless. Like I say, quick one today. Like, subscribe if you haven't already.