Large language models represent text using tokens, each of which is a few characters. Short words are represented by a single token (like "the" or "it"), whereas larger words may be represented by several tokens (GPT-4o represents "indivisible" with "ind," "iv," and "isible").
When OpenAI released ChatGPT two years ago, it had a memory—known as a context window—of just 8,192 tokens. That works out to roughly 6,000 words of text. This meant that if you fed it more than about 15 pages of text, it would “forget” information from the beginning of its context. This limited the size and complexity of tasks ChatGPT could handle.
It's been a really busy month for Google as it apparently endeavors to outshine OpenAI with a blitz of AI releases. On Thursday, Google dropped its latest party trick: Gemini 2.0 Flash Thinking Experimental, which is a new AI model that uses runtime "reasoning" techniques similar to OpenAI's o1 to achieve "deeper thinking" on problems fed into it.
The experimental model builds on Google's newly released Gemini 2.0 Flash and runs on its AI Studio platform, but early tests conducted by TechCrunch reporter Kyle Wiggers reveal accuracy issues with some basic tasks, such as incorrectly counting that the word "strawberry" contains two R's.
These so-called reasoning models differ from standard AI models by incorporating feedback loops of self-checking mechanisms, similar to techniques we first saw in early 2023 with hobbyist projects like "Baby AGI." The process requires more computing time, often adding extra seconds or minutes to response times. Companies have turned to reasoning models as traditional scaling methods at training time have been showing diminishing returns.
Google has released what it’s calling a new “reasoning” AI model — but it’s in the experimental stages, and from our brief testing, there’s certainly room for improvement. The new model, called Gemini 2.0 Flash Thinking Experimental (a mouthful, to be sure), is available in AI Studio, Google’s AI prototyping platform. A model card describes […]
Google DeepMind, Google’s flagship AI research lab, wants to beat OpenAI at the video-generation game — and it might just, at least for a little while. On Monday, DeepMind announced Veo 2, a next-gen video-generating AI and the successor to Veo, which powers a growing number of products across Google’s portfolio. Veo 2 can create […]
Google has released a prototype of Project Astra’s AR glasses for testing in the real world. The glasses are part of Google’s long-term plan to one day have hardware with augmented reality and multimodal AI capabilities. In the meantime, they will be releasing demos to get the attention of consumers, developers, and their competition. Along […]
Google is slowly peeling back the curtain on its vision to, one day, sell you glasses with augmented reality and multimodal AI capabilities. The company’s plans for those glasses, however, are still blurry. At this point, we’ve seen multiple demos of Project Astra — DeepMind’s effort to build real-time, multimodal apps and agents with AI […]
Last week, Google DeepMind announced Genie 2, a new video model that generates plausible, consistent, playable 3D environments based on a prompt image. DeepMind claims that Genie 2 has a slew of new and exciting emerging capabilities that improve the lighting, reflections, and can even generate videos from real-world images. Within these generated worlds, the […]
Google’s DeepMind team unveiled an AI model for weather prediction this week called GenCast. In a paper published in Nature, DeepMind researchers said they found that GenCast outperforms the European Centre for Medium-Range Weather Forecasts’ ENS — apparently the world’s top operational forecasting system. And in a blog post, the DeepMind team offered a more […]
Google DeepMind said GenCast, its new weather forecaster, can make better predictions.
NASA/Getty Images
Google DeepMind unveiled GenCast, its AI tool for weather forecasts.
GenCast outperformed existing forecasting systems in trials, Google DeepMind said.
Better forecasting will allow for better preparation in extreme weather events, the company said.
Google DeepMind announced its new artificial intelligence weather prediction tool called GenCast on Wednesday.
Google DeepMind said GenCast differs from other models because it has "adapted to the spherical geometry of the Earth and learns to accurately generate the complex probability distribution of future weather scenarios when given the most recent state of the weather as input."
As a result, Google DeepMind said GenCast provides better forecasts than the "top operating system," referencing the European Centre for Medium-Range Weather Forecasts and its model — known as ENS — that makes predictions up to 15 days in advance. GenCast, which was trained on the European center's data, "consistently outperformed ENS" when it predicted extreme heat, extreme cold, and high wind speeds.
"Now consider tropical cyclones, also known as hurricanes and typhoons. Getting better and more advanced warnings of where they'll strike land is invaluable. GenCast delivers superior predictions of the tracks of these deadly storms," the company said.
In an article in Nature published on Wednesday, Google DeepMind researchers wrote that GenCast had "greater skill than ENS on 97.2% of 1,320 targets we evaluated."
"Better predictions of extreme weather enable better decisions," DeepMind said in its press release. The company also said it would share real-time and historic forecasts from GenCast.
"We are eager to engage with the wider weather community, including academic researchers, meteorologists, data scientists, renewable energy companies, and organizations focused on food security and disaster response," the company said.
DeepMind, Google’s AI research org, has unveiled a model that can generate an “endless” variety of playable 3D worlds. Called Genie 2, the model — the successor to DeepMind’s Genie, which was released earlier this year — can generate an interactive, real-time scene from a single image and text description (e.g. “A cute humanoid robot […]
Google's DeepMind and YouTube previously built and shelved Orca, an AI music tool.
Orca could generate music mimicking artists. Google trained it on copyrighted YouTube music videos.
Google's AI strategy led to Orca's development. Legal risks halted its release.
Name your favorite artist, choose a genre, and feed it some lyrics, and AI will create a song that sounds completely authentic.
That was the vision of "Orca," a project Google's DeepMind and YouTube collaborated on and ultimately shelved last year after butting up against copyright issues, according to four people familiar with the matter, who asked to remain anonymous because they were not permitted to talk to the press.
The tool, which was internally codenamed "Orca," let anyone generate music with just a few simple prompts. It was developed as Google scrambled to catch OpenAI.
Users could generate a new song by giving Orca prompts like a specific artist, lyrics, and musical genre, said one person familiar with the project. For example, they could use the tool to generate a hip-hop song with the voice of Taylor Swift, that person said, adding that it was "mind-blowing."
Google eventually approached some music labels about releasing the Orca tool to the public, offering a revenue-share agreement for the music and artists Orca trained from, and the labels demurred, forcing Google to put the brakes on the project, that person said, adding that it was a "huge legal risk."
Orca is yet another example of how tech companies have moved at breakneck speeds to get ahead in the AI race. It also demonstrated how tech companies were willing to ride roughshod over their own rules to compete.
Google had previously avoided using copyrighted videos for AI training. When OpenAI started scraping YouTube for its own models, Google leadership decided to be more aggressive and reneged on its rule, said a person with direct knowledge of Orca.
Google has terms that allow it to scrape data from YouTube videos to improve its own service, although it's unclear if building an AI music generator would fall under this policy.
Developments on Orca throughout 2023 were so promising that at one point, some employees suggested that giving it a codename after a killer whale wasn't a good idea if DeepMind was about to destroy an entire music industry, one person involved recalled.
Some researchers inside Google had developed a similar model of their own, MusicLM, trained on "a large dataset of un-labeled music," as detailed in a paper published early last year.
In November 2023, DeepMind announced a music generation AI model named Lyria, which was a pared-down version of the Orca project. Users could ask Lyria to generate music using the voice and music style of some artists who had explicitly worked with Google on the project, such as John Legend — although it was far more limited in scope than Orca, three people familiar with the project said.
Some employees who worked on Lyria and Orca left to found a new startup named Udio, which makes an AI music creation app.
Google did not respond to a request for comment.
Are you a current or former DeepMind or YouTube employee? Got more insight to share? You can reach the reporter Hugh Langley via the encrypted messaging app Signal (+1 628-228-1836) or email ([email protected]).
