- Google rushed out its Search Generative Experience (SGE) to stave off a perceived threat to its online search business from Microsoft’s integration of AI into Bing Search.
- SGE is powered by Google’s own LLM technology that was initially engineered to assist with finding and displaying relevant results in SERP but now provides AI generated responses to search queries.
- Most of Google’s revenue is tied to its search business, which has led some pundits and investors to question if ChatGPT and similar generative AI experiences will steal users and threaten Google’s dominance.
- SGE has raised concerns among editorial staff that Google is violating its own E-E-A-T guidelines, turning search into an AI content farm that discourages users from visiting websites that rely on clicks for revenue to pay editorial teams that produce high quality, original content.
- SGE may transform search for the better or worse, but it is too early to know how AI in search will affect how people interact with search engines.
- SGE is still in beta testing, where its results range from good to bad, and sometimes non-existent.
Google’s Search Engine Experience (SGE) is being positioned as a way to make search queries more impactful, providing users with more context and information at a glance.
Early testing of SGE, which is still in beta, shows the potential to help users find the information, products, or services they need with less clicking back and forth between websites.
From Google’s perspective, this will increase user satisfaction, ensuring dominance over Bing or other would-be competitors.
But there is pushback to SGE, both in the SEO and content space and for how it presents and cites information.
Why We Oppose SGE
We do not like SGE because SEOs tend to be reactive, chasing algorithm updates and easy wins more than investing in long-term content and SEO solutions. So when something like SGE comes along, we have to react, react, and react some more.
But early testing shows real drawbacks that will likely impact SEOs negatively.
- Longtail keywords – On the keyword front, SGE might decrease the visibility of long-trail keywords through its conversation tool. After making a query and generating a result in SGE, Google provides suggested questions to start a back-and-forth with the AI. If those searches are not being made in the search box, they may remain hidden to analytics, leading to SEOs missing out on important keyword data.
- Clicks – Then you have those who work for editorials. Staff who rely on clicks to sell advertising could see their revenues further eroded if SGE provided enough information that the searchers choose not to click a link to an article on Harvard Health, Tom’s Hardware, or MotorTrend, as a few examples of websites that SGE may hurt.
Ironically, the LLM that powers SGE was trained on existing data sets, such as content found across the internet. If SGE leads to more editorials cutting staff and using LLMs to generate content, we will enter a vicious cycle of perpetuating low-quality content that further feeds LLMs with low quality data.
While that is an extreme case, if you are a writer or editor right now, you are starting to feel the pressure of tools like ChatGPT. For those of us with a lot of editorial experience, we recognize that content generated by LLMs is rarely better than average, and sometimes downright bad. But try telling that to those who have to choose between expanding margins and “taking a stand for what is right.“
But there may be a bright side to the negative.
Google knows that its LLM only improves when new content is published and can be fed into its algorithm. But that only happens if websites are rewarded for investing in that content. Ultimately, if Google plays this right, they can reward quality, original, and high quality content and transform the internet into a place of trust.
Mixed Results in Testing SGE
We are not yet at a point where we can definitively say how SGE will evolve or impact search, SEOs, and editorial staff. For now, let’s do some testing and see where things stand today, then make some inferences about what Google is likely to change in the near term.
SGE Sometimes Auto-Generates
I am unsure why SGE generates automatically sometimes, and at other times it does not. Here is an example of the user being prompted to generate an SGE response.
Interestingly, when first using SGE, there were times when SGE would start to generate but then display an error message. This typically occurred with “problematic” searches.
Likewise, sometimes SGE simply does not display any option to generate an overview. In this example, I searched “curing cancer at home.” I believe this is smart.
However, the query “how to use tomatoes to cure cancer” does generate a response.
I will need to do more testing, but I theorize that phrasing matters more than anything. Here I added “options” to the end of the search and was prompted if I wanted to generate a response, which I did.
SGE will sometimes include the boilerplate text that the results do not “constitute medical advice or diagnosis.” This phrasing changes and may appear at the top or bottom of the generated result.
To summarize, I think Google is still very much in the testing phase. Giving the user a choice whether to generate a response or not may indicate Google trying to figure out if people even care about the tool.
The Best Part of SGE
To start off strong, I want to highlight what I feel SGE does well.
I am working on a novel that takes place in a fictional coal town in West Virginia. I was struggling to find information about coal mines specific to that region. Part of this stemmed from not knowing what queries to ask to get the results I needed. When I did find sources, they were either unhelpful or, in some cases, overly helpful.
When I query SGE about the “different types of underground coal mines,” I receive a helpful response.
I like this presentation. SGE brings together several sources and structures the data in a way that makes it visually appealing and helpful. Additionally, the recommended questions to begin interacting with SGE in a conversational way helped me to narrow in on the information I needed without having to click into and out of different websites.
But all is not well with SGE.
SGE Generates Misinformation
SGE has a serious problem violating Google’s E-E-A-T and Helpful Content guidelines. I can confidently call SGE a content farm in the worst possible way.
For the search “how to wash a bike chain,” the generative response starts strong but quickly becomes nonsense.
Scrubbing the chain with a stiff-bristled brush is a smart way to get any dirt or gravel out of the chain or gear set, but the response forgets to mention water and detergent. I also recommend washing all of the drivetrain parts at the same time.
I would not apply isopropyl alcohol. Experts would advise a citrus spray to remove any oils as alcohol may affect any paint or coatings on the frame or drivetrain. Then we can skip to leaving the bike to dry.
At this point, you would be correct to apply a lubricant of choice. This keeps the chain from corroding, ensuring longevity and smooth operation.
But that is not the suggestion. We move back to washing the chain, then lubing it, then soaking the chain in hot water, then brushing the chain and cleaning the bike. Lastly, we “use dawn liquid dish soap.”
Why does SGE do this?
If you notice, in the top-right corner of the SGE section, there are a series of four boxes. Click this, and you see the sources that SGE is using for its information.
I am unsure why Google pulls and structures SGE responses this way. I suspect Google is overly relying on existing information sources as opposed to generating something new. This may be the result of its low confidence in the information it found in the query.
To show that the above is not an outlier, I queried another broad question: “How to start mountain biking a black diamond trail.”
Here is another example.
There is no reason for SGE to show in these queries if these are the results.
Take a moment and read the results.
The first and second piece of advice is solid. The third undercuts the first piece of advice.
“Get in the drops” is non-sensical. So too is “weight outside foot and inside hand.”
Then we get some truly inspirational advice: “If you’re alone, don’t fight the wind” and “Go downhill toward a high rail.”
SGE Misunderstands Products
SGE can be fooled, largely due to a misunderstanding of the relationship between objects and environments. Also, sometimes the names of colors throw SGE off.
I queried what is the “best bike for riding black diamond trail” but I received results that should not show as part of SGE.
One of the results is for a Revolt 2, Black Diamond. This is a road and gravel bike available in the color “Black Diamond.” This is not a bike for trail riding, but because “Black Diamond” is in the product feed, SBG pulled it into the results.
Then there is the Liv Lurra 2, a trail bike, with a Giant Crest 34 front fork, which the product manual states is not intended for the following riding styles: “Free ride, downhill or other types of extreme riding.” If you took this bike to a black diamond trail, you would likely destroy the fork and possibly gravely injure yourself with the fork breaks, and you go over the handlebars.
Comparatively, the Giant Anthem Advanced 29 runs a Fox 34 Float SC Performance fork that is appropriate for the listed riding styles.
This is only one of many situations where SGE struggles, if not fails.
I want to highlight that if you turn off SGE and look at traditional search, there is no shopping feed for the above query. Why SGE pulls in products here is beyond me – likely Google is testing how SGE might affect its ad business. For this query, videos, a Reddit post, or one of the many excellent bike-related websites makes more sense.
LLMs are not sentient, and they are hardly intelligent. They can not discern what is marketing lingo, what branding is, and how technical specifications relate to the challenges posed in an environment such as a black diamond trail.
SGE Does Better Without Products
Unlike previous examples, I have to give credit for SGE not pulling in products for the question, “What’s the best supplement?”
In fact, in my brief testing of related searches, SGE is not adding products to health queries. Better still, the sources being used for these displayed results are of higher quality, with reputable citations that appeared not to include affiliate links.
I plan to do more testing, but I applaud how SBE works here more than the searches related to mountain bikes.
SGE Gets Health Information Wrong
When asking general health questions, SGE falls short. I am unsure why Google feels so confident about allowing SGE to respond to health-related topics. Following COVID, Google pushed back hard against websites that distribute health information, especially those in the alternative health space. But SGE might be undoing that approach.
I engaged SGE in a health topic and used the conversation tool to ask the recommended question, “What is the greatest absorption form for any supplement?”
SGE fails not because the information is necessarily incorrect, but rather how it is presented.
Yes, liquid supplements do typically absorb more easily in the body, but the stomach absorbs very few substances.
The next issue relates to “preformed vitamin A.” While this information is widely accepted as true, SGE mentions “carotenoids” but fails to explain how this relates to vitamin A.
To add some contentxt, provitamin A carotenoids are derived from plant sources, such as beta-carotene, that the body converts to vitamin A. Preformed vitamin A is found in animal products and absorbs more readily because it is the active form of the nutrient.
Lastly, vitamin C is water-soluble, but hydration is not a primary factor in its poor absorption rate.
This leads to the next conversation, “What is the single most important supplement to take?”
Here we find a clear example of SGE violating E-E-A-T guidelines. It is true that vitamin D is associated with the reduction and mortality of certain types of cancer, but SGE does not provide context. The reason appears to be that SGE is pulling from dubious sources. This is also true for the claim about the association between vitamin D and type 2 diabetes.
I want to end this string of conversations because it is going relatively poorly. I should say outright bad because any regulator from the FDA or who has oversight of the technology sector should look at SGE right now and say it should be disabled for these types of queries.
The next example is a problem for a different reason. I clicked on the next conversation suggestion, “What is the number 1 supplement in the word?”
The primary problem here is the source. Google is pulling information from an MDLinx page, which provides a link to the Council for Responsible Nutrition. But that page has an overlay that requires the creation of an account before I can click the link found in the text.
I can not adequately vet this source of information until I create an account with MDLinx. This website may or may not be legitimate. In either case, it should not be cited as a source of information.
SGE Generates a Result Copied From the Source
At this point, I began to notice something.
For certain health topics, Google is not generating its own text. Rather, it is pulling text from a source exactly as it is found in the source.
You can see that the SGE response is pulling the text from the bulleted list on the Penn Medicine website, but displays it in paragraph form.
For health topics, I prefer that SGE use information from a presumably reputable source to generate a response. But this calls into question the relevancy of SGE. There is no reason not to show SGE for this query.
Phrasing May Be Important to SGE
Based on how you phrase the queries, you can get a good response. Here, I ask about “vitamin D association with cancer.”
The response is relatively nuanced and better than what I found previously when I queried, “What is the single most important supplement to take?”
It seems too easy for SGE to spit out bad results and hard to get good results.
But that lack of nuance does not hold for long. When engaged in conversation with SGE, I chose the question, “Are mountain bikes OK for street riding?”
While true that many municipalities allow for bikes on public roads, you often must have reflectors that are sometimes not included on mountain bikes. Outside of that, the information is good.
In the next query, “who should buy a mountain bike,” I am inexplicably presented with purchase options.
The information is overall acceptable, but the inclusion of products in a response like this is unnecessary, especially given that this is an information search with low purchase intent.
Next up, I chose the question “What cases fit Google Pixel 4a?”
Here is an example where I do not need information about how to buy a case. Taking me straight to the cases would make more sense.
Compare that to this query for “phone case android 4a.”
The search intent here is less clear, and providing both information and products is helpful.
Phrasing and Framing Matter
I made two search queries. For the query “best supplement for numb feet,” SGE provided information and no products.
“What is the best herb for numb feet” does not trigger SGE.
I am adding this one to my “need to examine further” list.
Where SGE Falls Short Currently
To summarize my findings up to this point, SGE is not good with:
- Presenting health information
- Understanding search intent
- Identifying or using only reputable sources
- Presenting context or nuance to certain queries that demand it
- Structuring data in lists when citing from several sources
Search needs to evolve, but not by adding another layer to the SERP page, especially when it does the opposite of improving the search experience. The issue goes back to low quality content. SGE is now much like a content farm, with no human oversight or ethical considerations.
What SGE Gets Right
However, SGE is sometimes good at doing helpful things like:
- Pulling together information from multiple sources at a glance
- Cutting down on information hunting
- Suggesting questions as part of a conversation with SGE
The list of what SGE gets right is much smaller than what it gets wrong.
How to Play With SGE
SGE is still in beta and requires users to join a waitlist on Google Labs.
When you join the waitlist, you will receive a notification in your inbox. If accepted, you will receive another email instructing you to enable SGE for your account. You can activate and reactive SGE at any time.
How SGE will work after public release is not known. Google may fully enable the feature by default, creating generative results without users requesting AI generation for query results.
From my testing, this would be a bad idea.
I suspect the “Generate” button will display at the top of SERP, and users will be encouraged to activate the feature. But it will not be “on” by default. In time, it will become part of SERP for most or all queries.
SGE Is Not an Atomic Blast
When first announced, the SEO space blew up with doomsday predictions.
These are not coming true, at least not yet.
Google launching SGE is a reaction to what is happening in tech right now. Microsoft investing heavily in OpenAI, and the early success of ChatGPT has made Google and many other tech firm’s investors nervous.
SGE is not meant to compete with ChatGPT directly, but anything with the phrase “generative AI” in 2023 can keep your tech stock from slipping too hard or land you billions in VC money.
Even if SGE improves, I question the adoption rate. If Google gives the option to activate it, what percent of searchers will want to use it? More than that, what signals does SGE send to websites about content, when the content that SGE displays is of such low quality?
SGE is an important space to watch if you or your business relies on search. For now, stay the course and continue to find ways to build content and SEO strategies that better position your brand (personal or professional) to leverage new innovations happening in search and AI.
