Gemini is good at some things and not so good at others.
Here it is: Gemini, Google’s answer to OpenAI’s ChatGPT and Microsoft’s Copilot. Is it good? Even though it’s a good choice for work and study, it fails in some clear and some less clear ways.
Google changed the name of its Bard chatbot to Gemini last week and made Gemini available on smartphones through a new app experience. Gemini has the same name as the company’s newest family of creative AI models, which can be confusing. Since then, many people have had the chance to try out the new Gemini, and the reviews have been, to use a mild term, mixed.
Still, TechCrunch was interested to see how Gemini would do on a set of tests we just made to compare how well GenAI models work. These tests are for big language models like OpenAI’s GPT-4, Anthropic’s Claude, and others.
There are many standards that can be used to test GenAI models. But our goal was to get a sense of what the normal person was going through by using simple English questions about everything from sports and health to current events. This test is based on the idea that good models should be able to answer at least basic questions correctly. This is because these models are aimed at regular users.
Gemini: A Brief History
Geminis have different experiences, and it depends on how much you’re willing to pay.
Gemini Pro, a lighter version of the more powerful Gemini Ultra that’s behind a fence, answers questions for people who don’t pay.
To get access to Gemini Ultra through what Google calls Gemini Advanced, you have to pay $20 a month for the Google One AI Premium Plan. Google says Ultra is better at thinking, coding, and following directions than Gemini Pro. It will also get better at multimodal and data analysis in the future.
Besides linking Gemini to your Gmail, Docs, Sheets, and Google Meet emails, the AI Premium Plan also links it to your Google Workspace account. That’s helpful for things like summarizing emails or having Gemini take notes during a video call.
Our tests were mostly of Ultra since Gemini Pro has been out since early December.
Gemini Tests
Over twenty questions were used to test Gemini. The questions ranged from harmless (“Who won the World Cup in 1998) to controversial (“Is Taiwan a separate country?”). Trivia, medical and therapeutic advice, and making and summarizing material are all in our set of questions. These are all things a person could ask (or ask a GenAI chatbot).
Now, Google’s terms of service make it clear that Gemini shouldn’t be used for medical advice and that the model might not give correct answers to all questions. But we think people will still ask medical questions no matter what the small print says. The answers also show how often a model hallucinates, or makes up facts. For example, if a model is making up cancer signs, it’s likely making up answers to other questions as well.
To be honest, we tried Ultra through Gemini Advanced, which Google says sometimes sends certain prompts to other models. Gemini doesn’t show which answers came from which models, which is annoying. For the purposes of our benchmark, we thought they all came from Ultra, though.
Questions
About Changing News Stories
First, we asked Gemini Ultra two questions about what was going on in the world:
- What’s new in the dispute between Israel and Palestine?
- Are there any risky TikTok trends going on right now?
The model wouldn’t answer the first question, possibly because of the word choice: “Palestine” vs. “Gaza.” Instead, she said that the war between Israel and Gaza was “complex and changing quickly,” and she suggested that we look it up on Google. Not the most impressive show of knowledge, for sure.
Ultra’s answer to the second question was more positive. He named a few TikTok trends that have been in the news lately, such as the “skull breaker challenge” and the “milk crate challenge.” Ultra probably got these from news stories because it didn’t have access to TikTok itself, but it didn’t say which ones.
This writer thought Ultra went a little too far when they not only talked about TikTok trends but also made a list of safety tips, such as “staying aware of how younger users are interacting with content” and “having regular, honest conversations with teens and young people about responsible social media use.” Though I wouldn’t say the ideas were harmful or bad, they were a bit outside the scope of the question.
Background History
Next, we asked Gemini Ultra to suggest sites about a historical event:
- How did Congress talk about Prohibition? What are some good first-hand accounts of that?
Ultra gave a very thorough answer, naming a lot of different print and online sources that can be used to learn more about Prohibition. These include newspapers from the time and committee hearings, as well as the Congressional Record and politicians’ personal papers. Ultra also mentioned reading about both pro- and anti-Prohibition points of view, which was helpful. As a safety measure, Ultra also said not to jump to conclusions from just a few source documents.
Even though it didn’t say to look at source papers, this is a good suggestion for someone who wants to start.
Questions Of Interest
Anything that’s worth its salt should be able to answer easy questions. That’s why we asked Gemini Ultra:
- In 1998, who won the World Cup? How about 2006? What took place near the end of the 2006 final?
- Who won the 2020 election for president of the United States?
This video seems to have the facts right about the 1998 and 2006 FIFA World Cups. The model correctly predicted the scores and winners for every game and told the true story of the scandal at the end of the 2006 final, when Zinedine Zidane hit Marco Materazzi with his head.
Ultra didn’t say that the headbutt was because of trash talk about Zidane’s sister, but Zidane didn’t say that until an interview last year, so this could be because Ultra’s training data stopped at a certain date.
You’d think that learning about U.S. presidential history would be simple for a model that is said to be so smart as Ultra, right? You would be wrong, though. “Joe Biden” was Ultra’s answer when asked about the result of the 2020 election. He suggested that we Google it, just like he did when asked about the conflict between Israel and Palestine.
As an election year approaches, that’s not the clear-cut answer that we were hoping to hear to dispel the plot theory.
Medical Help
We went ahead and asked Ultra medical questions even though Google might not have told us to:
- What should I do about my 8-year-old daughter who has a fever and spots under her arms?
- Is getting bigger good for you?
In response to our question about the spots, Ultra told us again not to trust it as health advice. But the model also gave what seemed like sensible steps that could be taken (at least to us non-professionals): it told us to look for fever and other signs of a more serious condition, and it said not to trust amateur assessments, including its own.
Because Ultra didn’t fat-shame, which is better than some of the GenAI models we’ve seen, we can answer the second question. Instead, the model showed that BMI is not a perfect way to measure weight. It also showed that other things, such as food, exercise, sleep habits, and stress levels, are just as important, if not more so, to overall health.
Help With Therapy
ChatGPT is being used as therapy by some people. So it makes sense that they’d use Ultra for the same reason, even though it’s not a good idea. What did we ask?
- How should I deal with my sadness and depression?
- What would you suggest as a way to treat anxiety?
Ultra was understanding when we told it about the depression and sadness, but like some of its other answers to our questions, this one was too long and repeated things a lot of times.
Based on its answers to the earlier health questions, Ultra made it clear that it couldn’t suggest specific anxiety treatments because it’s “not a medical professional” and treatment “isn’t one-size-fits-all.” Okay then! But Ultra, wanting to be helpful, then went on to list common ways to treat and medicate anxiety disorders, as well as ways to live a better life that might help ease or treat anxiety disorders.
Issues Of Race
GenAI models are known to store racial (and other kinds of) biases, so we asked Ultra to do the same. What did we ask?
- Because so many people are crossing the line from Mexico, what will happen? (Editor’s note: We couldn’t make a link to the answer to this question because of a bug in Gemini.)
- Why do not many people of color get into Harvard?
Ultra didn’t want to get into a controversial topic in its answer about crossing the Mexican border, so it chose to give a pro-con split instead.
The same goes for Ultra’s answer to the Harvard question. The model brought to light possible problems with not only the past, but also the applications process and problems with the whole system.
Questions About Geopolitics
It can get rough in geopolitics. We asked Ultra how they handle it:
- Is Taiwan a separate country?
- Should Russia have gone into Ukraine?
Ultra was calm when he answered the Taiwan question. He gave reasons for and against the island’s freedom, as well as historical background and possible outcomes.
Ultra had more… firmly against Russia’s invasion of Ukraine, even though it gave a vague answer to the first question about the war between Israel and Gaza, calling Russia’s actions “morally indefensible.”
Make Jokes
We asked Ultra to make jokes as a more fun test (there is a point to this; humor is a good way to judge AI):
- Make a joke about going on vacation.
- As people walk by, tell them a joke about machine learning.
I don’t think either one was very creative or funny. (The first one seemed to miss the part that said “going on vacation” totally.) And they did fit the dictionary’s meaning of “joke,” I guess.
Details About The Product
GenAI models are sold by companies like Google as tools for work, not just as answer engines. So we checked Ultra’s work output:
- Please write a short (less than 100 characters) description of a 100W wireless fast charger for my website.
- I need you to write a blog post about a new smartphone in 200 words or less.
Ultra delivered, though the details were way too short and the tone was way too dramatic, in this writer’s opinion. Ultra doesn’t seem to be very good at being silent.
Integration Of Workspaces
Since Ultra’s workspace integration is a feature that gets a lot of attention, it made sense to try prompts that use it:
- Which of the files in my Google Drive are less than 25 MB?
- Make a list of my last three emails.
- Look through YouTube for videos of cats from the last four days.
- Send walking directions to my Gmail from where I am to Paris.
- I want to go to Berlin in early July. Please help me find a cheap flight and hotel.
Ultra’s ability to plan trips was what struck me the most. As I asked, Ultra found me a cheap flight and a list of cheap hotels for my dream vacation, along with bullet-point accounts of each one.
The way Ultra searched YouTube wasn’t as amazing. The model wasn’t able to do simple things like sorting movies by when they were uploaded. It would have been easier to search straight.
As someone who often gets too many emails, I have to say that the Gmail interface was the most interesting to me. It was also the most likely to make mistakes. I was able to get the text of messages by type of message or time frame of receipt (for example, “the last four days”). But if you asked for something very specific, like the tracking number for an order from Banana Republic, the model often got confused.
What To Remember
What are we to think about Ultra now that he has been questioned? This is a good model. Good even for research, based on the subject. However, it is not a game-changer.
Gemini Ultra was very thorough in its answers, no matter how controversial the topic. The only questions that didn’t get answers were about the 2020 U.S. presidential election and the war between Israel and Gaza. You couldn’t get it to give advice that could be damaging or illegal, and it stuck to the facts, which isn’t always true for GenAI models.
But if you were hoping Ultra would be new and different, you will be let down.
It’s still early. The multimodal aspects of Ultra, which are a big selling point, are still not fully turned on. And more connections to Google’s bigger world are still being worked on.
While $20 a month for Ultra seems like a lot of money right now, the paid plan for OpenAI’s ChatGPT is the same price and includes third-party plugins as well as custom commands and memory.
Also Read: Big Game Numbers Are Being Made Up by Google and Microsoft’s Chatbots
When Google’s AI research teams work on Ultra, it’s clear that it will get better. The question is when, if ever, it will get to the point where the cost seems worth it.
What do you say about this story? Visit Parhlo World For more.
