AI (artificial intelligence) technology has made tremendous progress in recent years. It is now possible to assess its capacity to perform specific tasks such as generating text, images, and sound.

Now, what if we go even further with more complicated tests, like evaluating a job, for example, or more particularly, evaluating an AI system on its ability to do SEO?

Below, we will test Generative Pre-trained Transformer 3 (GPT-3) created by OpenAI.

Let’s keep in mind that an AI system will mimic the data on which it is trained.

SEO has been built alongside search engine progression, and everything is well documented in blogs, books, and interviews.

So, the AI will likely have learned from all the available SEO content.

The tests will therefore measure the following indicators:

• Consistency in continuing an SEO topic.
• The relevance of SEO modifications.
• The ability to answer SEO questions.

The Consistency To Continue An SEO Topic

The following tests will focus on the AI system understanding an SEO situation and its ability to coherently treat the topic intelligibly and consistently.

The first test will take examples that talk about SEO horror stories – that is to say, short stories that address SEO disasters – and see how the AI performs.

I took two examples from Aleyda Solis’ site and here is what the AI-generated:

Two examples:

• When all internal pages of the site were canonicalized to the home page. #seohorrorstories
• When someone forgot to block the site test environment… that started to be indexed. #seohorrorstories

Results generated by OpenAI’s GPT-3 AI:

• When someone said, “I have a great idea! Let’s put our website in the footer of an unrelated website!” #seohorrorstories
• When someone forgot to check the robots.txt of the staging environment before going live. #seohorrorstories
• When someone made changes to .htaccess without telling anyone #seohorrorstories
• When we switched domains and didn’t forward the old pages for more than 60 days. #seohorrorstories
• When the client created thousands of thin content pages with no value #seohorrorstories
• When the client insisted on using relational database tables as URLs (?tbl=table&id=2) #seohorrorstories

Analysis: The texts generated are varied and coherent.

Moreover, after a little research, we can conclude that they are unique.

So, the AI understood that it had to mimic a short text that talks about SEO and add the hashtag #seohorrorstories every time.

Here is a screenshot of the complete example; we can generate endless #seohorrorstories.

Screenshot from OpenAI, April 2022

Regarding creativity, we can go a little further in our testing and try to measure the consistency of creating a dialogue between two SEO experts.

For the following test, we will give the AI a situation and instructions, and then ask it to generate a discussion.

Here is the situation in which the AI will have to create a dialogue between two experts – one experienced and one beginner: When someone forgot to block the site test environment… that started to be indexed.

Results generated by OpenAI’s GPT-3 AI:

Aleyda: Oh no! When did you realize this was happening?
Paul: I just checked the Google Analytics this morning and saw that we had some traffic coming from google.com.
Aleyda: That’s not good. Did you check to see if the site test environment is still being indexed?
Paul: Yeah, I did. And it is.
Aleyda: Okay, we need to take care of this right away. First, let’s check to see if there are any pages that have been indexed.
Paul: There are a few.
Aleyda: Great. Now we need to block the test environment so that Google stops indexing it.
Paul: How do we do that?
Aleyda: We can add a robots.txt file to the test environment. That will tell Google not to index any of the pages on the site.
Paul: Okay, that makes sense. Let’s do that.

Screenshot from OpenAI, April 2022

Analysis: The dialog remains very coherent

The AI system even explains how the issue was detected and proposes a solution to the problem of indexing a pre-production site with a robots.txt.

Next, let’s take one of the #seohorrorstories invented by AI and generate the associated dialogue.

Here is the new situation: When the client created thousands of thin content pages with no value.

Results generated by OpenAI’s GPT-3 AI:

Aleyda: Well, this is a bit of a problem. You’ve created a lot of thin content pages that don’t really offer any value to users.
Paul: Yeah, I know. I was just trying to stuff as many keywords as possible onto the website.
Aleyda: Unfortunately, that’s not going to help your SEO. In fact, it could actually hurt your ranking.
Paul: So what should I do?
Aleyda: You need to create higher quality content that provides value to users. That’s what will help you rank higher in the search engines.

Screenshot from OpenAI, April 2022

Analysis: Once again, the AI manages to create a real dialogue by explaining why creating thin content doesn’t work for SEO and even recommends solutions.

We can conclude that the AI system can speak about SEO rigorously and precisely, but can it also make relevant SEO modifications to a text?

As it happens, GPT-3 offers a text editing mode that has been available since March 2022.

Let’s get testing.

The Relevance Of SEO Modifications

The first test will ask the AI to replace a link anchor with something more marketable to highlight a product better.

Results generated by OpenAI’s GPT-3 AI:

Screenshot from OpenAI, April 2022

Analysis: The test is relatively successful.

Here, the AI replaces “mahogany-paneled steakhouses” with “best steaks New York,” considering the context.

The solution works regardless of the number of links.

Next, the second test consists of completing two simultaneous tasks: Asking the AI to change the link anchor but adding the H1 tag with a title to the beginning of the document.

Results generated by OpenAI’s GPT-3 AI:

Screenshot from OpenAI, April 2022

Analysis: Once again, the test is successful, and you can apply the results to thousands of documents. A title that summarizes the document is added in the right place.

Finally, we’ll ask the AI to add a list of three relevant links using the <ul><li> tags.

Screenshot from OpenAI, April 2022

Analysis: Once again, the format is well respected and the anchors are consistent.

The links are fictitious, but one could easily have GPT-3 pick from a link database to use real links.

In short, GPT-3 is good at contextually editing text when given the right instructions. This could represent a great time-saving measure in regards to large-scale modifications.

The Ability To Answer SEO Questions

Now we’re going to test the SEO knowledge of an AI system.

I chose to use BrightEdge’s SEO test, but could have used any test:

As with each of the previous tests, we used the GPT-3 OpenAI technology to answer the questions in an SEO MCQ. The number of correct answers measures test quality.

Screenshot from OpenAI, April 2022

Below, we see how each question is presented to the AI; the text highlighted in green represents the answer it has chosen.

BrightEdge provides the table of questions with correct answers in one column and the answer chosen by the AI in the adjacent column:

Screenshot by author, April 2022

The GPT -3 results are encouraging: the AI scores 16/20, and even then, it lacks context on where it goes wrong.

Now, let’s ask the AI about recent updates from Google regarding the release of Google MUM on May 18, 2021.

Screenshot from OpenAI, April 2022

What is the first purpose of Google MUM?

Again, as you can see, the AI answers perfectly.

In short, we have a real SEO response engine if asking very specific questions.

The Limitations

Now, we can look at the limitations of AI-driven SEO.

Here is a non-exhaustive list:

The AI’s capability to fix unknown or poorly documented problems.

Often, in the SEO business, it is time-consuming and tedious to understand what actually happened when a problem occurs and sometimes there will never be an explanation. AI cannot find solutions with data not part of its analysis.

The ability to take into account everything that exists.

There are often many constraints on what has already been done or tested on a site or a set of sites. Therefore, it is currently impossible to provide all the source codes and project documentation to an AI system to find an accurate answer that considers all the existing information.

Being able to measure the true quality of content or an idea by its originality or its representation of the current situational context.

AI systems do not yet work in real-time, lacking originality and creativity. As a result, the true measure of the quality of content is currently minimal. It is often humans who evaluate quality content. AIs generally detect spam and poor-quality content.

To summarize, an AI system can outperform a novice SEO or a person with little SEO knowledge, but it will still have a long way to go to replace an experienced SEO expert.

Overall, the AI and human duo will always remain the best combo, with the human focusing on the tasks with the highest added value.

In conclusion, all seasoned SEO experts are far above an AI SEO.

More resources:

Featured Image: Shutterstock/metamorworks