I’ll start out each of these by disclosing whether or not I have any relationship or financial interest in the company I review. Being a brilliant investor, when I had a little money to invest in the market a decade ago, I decided to put it into Twitter.com instead of Facebook.com or Google.com. Instead of owning a few islands in the Caribbean, I ended up selling my TWTR shares at a loss shortly before Elon Musk took it private. Today, I have no financial interest in Twitter or X, and I don’t own a Tesla (yet).

In SEMRush’s June 2024 “Winners and Losers” report, YouTube.com once again showed up as the biggest loser, shedding about 800 million keywords and losing 6.6 million visitors in SEO traffic.

But no need to cry for them. As I explained in my last post, this is a continued correction. As with YouTube (and others on the “losers” list like Wikipedia, Quora, and Reddit”, Twitter enjoyed a healthy boost in SEO traffic in early 2023 as Google started to steal from the poor (niche sites) and give to the rich (established sites with UGC).

I see more of the same here. This slight downturn is really just a correction of Google rewarding UGC sites in its recent core algo updates. I go into a lot more detail in my Search Engine Land article.

The difference between YouTube and Twitter is that while Twitter’s visibility is up to over 200 million keywords and 3 billion organic search visits, YouTube’s has gone up to 3 billion keywords and 9 billion organic search visits (!)

This is a result of Google buying two things: YouTube in 2006 and the Federal Trade Commission in 2013.

Okay, I’m joking, but if Google wiggles their way out of the current anti-trust trial without any action taken against it…maybe it won’t be a joke after all.

Why doesn’t Twitter have as many visits as YouTube? The most obvious reason is that the type of content on YouTube is more often searched than the type of content on Twitter. More people will search for “how do I tie a tie” than “what random thought did so-and-so celebrity have a few minutes ago”.

Another reason is, to put it bluntly, Google owns YouTube. There are plenty of video hosting sites with lots of great content, but YouTube always seems to come up top. I wonder why.

All that said, Twitter is also hurting itself by committing a few very basic mistakes with their SEO. I’ll provide details below.

Why do you keep saying Twitter…I thought it was called X now?

Yes, and that’s going to be the point of the rest of this post.

I’ve been using “Twitter” because that’s what’s still showing up on Google, even though as of this writing it’s been almost exactly a year since Twitter rebranded to X.

X has done a fairly good job of pushing this change from a comms perspective. They’ve updated all their on-site copy to reflect the new name. Sites like the New York Times are dutifully using the phase X consistently and when characters like Stephen King refuse to change, Elon Musk jumps in to gently remind him.

Having said that, as of this writing it’s been almost exactly a year since Twitter rebranded to X. And yet I still see Google results for Twitter.com. Why?

The screenshot from SEMRush above tells you what’s going on with the Twitter.com domain. Here’s what’s been going on with traffic to the X.com domain.

So why does Twitter.com still have traffic from 2.5 billion while X.com is struggling to achieve 25 million?

It’s because X is committing a bunch of unforced SEO errors. This is pretty typical among companies filled with engineers who all have some knowledge of SEO but no dedicated SEO person.

What unforced errors is X making with their SEO?

If I were doing an audit of X’s SEO, here are the first things that I’d recommend to them. By the way, if you’re on the X team, feel free to take this advice and run with it.

1. Twitter.com URLs are not properly canonicalized or redirecting to X.com.

A lot of SEOs have already made the observation that X is 302 redirecting URLs on Twitter.com to X.com. Surely enough, you can see the HTTP header response if you type a Twitter.com URL into your browser.

That said, only a true SEO geek will realize that X is actually cloaking for Googlebot. The result is that all twitter.com URLs, as of this writing, are continuing to resolve with a 200.

Worse, the canonical tags all still reference twitter.com.

Put another way, as far as Googlebot is concerned, twitter.com and x.com are two completely separate sites, each with distinct rel=canonical tags, and both with content duplicating each other.

As with any duplicate content situation, Google has to choose between one or the other. Twitter.com, being by far the most authoritative domain of the two, continues to get the lion’s share of indexation.

This explains why Google has been so slow to index X.com URLs, even those that were created and linked after the name change.

Here’s what X’s engineers need to do:

• Good option: Try putting a proper rel=canonical tag on twitter.com URLs.
  
  I understand why you’re skittish about throwing away the twitter.com domain. If you wish to continue to cloak twitter.com URLs for Googlebot here’s something you can do right away with minimal risk:
  
  • Change the rel=canonical tags on the twitter.com domain to their equivalent x.com versions. That’s what the rel=canonical tag was intended to do.
  • You should see Google starting to deindex twitter.com URLs and index x.com URLs. While nothing is ever sure with Google, you shouldn’t see a big decrease in rankings or volume.
    
• Better Option: Bite the Bullet and Put a 301 Redirect in Place Already
  
  Having said all that, here’s the situation:
  
  Your boss’s boss wants the Twitter name to permanently go away. You’re sending multiple mixed messages to Google by having Googlebot resolve to twitter.com with a 200 and by having browsers 302 redirect to x.com.
  
  Also, I am not a big fan of cloaking. That’s against Google’s documented guidelines and asking for trouble.
  
  So at some point you’re just going to have to bite the bullet and do this:
  
  • Issue a 301 redirect from all twitter.com URLs to their equivalent x.com URLs and show this 301 redirect to both browsers and search engines.
    
  • Inform all your users to change any legacy links that say twitter.com to x.com.
    
  • Make sure there are no longer ANY references to twitter.com in your sitemaps, structured data, or internal links.
    
• Will there be pain? Yes, probably. Google no longer passes full link authority through a 301 as they used to, so you’ll probably see a dip in organic search traffic. But unless you can convince Elon to change the name back to “Twitter”, you’re just putting off the inevitable.
  
• July 31, 2024 Update: Today I noticed another anomaly. If I’m a logged-in user and go to twitter.com, my URL stays twitter.com. This is another miss.
  
  The X Engineering team may be saying to themselves, “why should I bother 301 redirecting logged in users, since Google will never be logged in?”
  
  But that’s forgetting that many of those logged in users will be sharing their URLs on blogs, other social channels, and back to X. That’s a lot of links that aren’t going to be directly counted towards x.com.
  

2. Improve your Title Tags

This is a silly bit of advice that I always give which always surprises people as to how effective it is. I’m updating it specifically for X, knowing what capabilities they have access to.

Your title tags currently look like this:

<title>SteveLiuSEO on X: "Thanks to @sengineland for publishing my latest piece on online forums https://t.co/No7gjX27C7 If you're a little skeptical of what I wrote, I did a little more analysis after I submitted this piece. Check out a few examples of what I found on this thread. (1/4)" / X</title>

Here’s what you need to do:

• Instead of duplicating the post in the Title Tag, use AI to write a proper title tag.
  
  Hint: you own Grok.
  
  
  
• Arrange the title tag so that the specific information is at the front, the user’s handle is in the back.
  
  You’re currently committing the classic mistake of putting general, repeated information in the front of the title tag and the unique information at the end.
  
  <title>Exploring the Resurgence of Online Forums: Insights and Analysis from @sengineland | SteveLiuSEO on X</title>

3. Fix your robots.txt

Your robots.txt is a bit of a mess as well.


Where do I begin?

• Your “Allow” statements for Googlebot are superfluous. You can’t use regular expressions in robots.txt file.
• The syntax of Disallow: * is nonsense.
• Disallowing URLs like /account/deactivated which no longer exist is just a waste of space (I have some thoughts on how you do handling for that URL, but that’s an entirely different conversation).
• Your boss’s boss probably wants you to block the GPTBot user agents, but I don’t see that in here.

I could go on. This whole robots.txt file is just a jumbled mess of lines that various engineers have put into it probably since Twitter began without really understanding how robots.txt works.

Get someone who understands SEO to look at your server logs and determine which URLs Googlebot is crawling that you don’t want it to crawl.

4. Fix your sitemaps

Your robots.txt file on x.com still references your sitemap at https://twitter.com/sitemap.xml.

When I open the sitemap at that URL (I have to do so as Googlebot, as you’re redirecting regular users), I see a ton of URLs on the twitter.com domain. Going back to point #1 above, you need to change these to X.com.

If you want Google to start indexing x.com URLs, in addition to changing canonicals you need to update your sitemap.

I haven’t had a chance to drive too deeply into the sitemap, but for a site like yours you definitely don’t want Google getting the firehose. You need to be tailoring your sitemap so that only the most noteworthy accounts and posts appear in it.

5. Fix your structured data

Your structured data is a bit sloppy. When I validate it against Google’s rich snipper tester, I see missing fields and critical issues. You’re going to want to clean that up.

6. Stop relying on Google for traffic.

This is by far the most important advice I have for X.

This only way this can be achieved is by setting up Grok as a Google alternative and getting your users to adopt it. Fast.

When Grok reaches the point that it can serve as a credible general answer engine for the top Google queries, open it up to the public—no sign-in, no paywall. In the AI Wars you need first-mover adoption before all else. OpenAI will be there quick, so you have no time to lose.

Heck, I’ll even design the UI for you.

Conclusion

Obviously I don’t have the time to do a full SEO audit for Twitter and X, but these are fairly typical of unforced errors that big sites have.

There’s a funny phenomenon that happens in Enterprise SEO. It would take someone like me all of 30 minutes to go through and fix their robots.txt file. It would probably take two hours of a programmer’s time to add a rel=canonical tag to twitter.com.

But the way it works, engineering departments usually have huge backlogs of projects to be done. Projects that tend to get prioritized are ones where there’s a clear revenue benefit. Projects like “fix the robots.txt file” and “add a canonical tag” tend to get eternally pushed the the bottom because you can’t “prove” the ROI.

I do hope that someone at X sees this and acts on it—these fixes alone, executed properly, will probably be enough to accomplish Elon’s directive with minimal risk and downside. They’re welcome to reach out to me via my Contact info if they want more advice.