But let’s leave our bipolar guesswork aside for a while, so we can concentrate on a more interesting issue. Namely, that it’s great that Google.cn is going to disappear, and that whatever happens to the rest of the Gs, the Chinese internet will be a better place when Google.cn is gone. Let’s start with some crude survey work:

Baidu, Google.cn or Google.com?

I improvised a little survey today in the office, where I asked three of my young Shanghai colleagues which search engines they like to use. Interestingly, the answers were very similar, and all included some form of the following statements:

• Baidu.com is better for local information and Chinese culture.
• Google.cn we use sometimes for international information.
• Google.com? Nah, that’s for foreigners.

These results are surprising, because as we saw yesterday, Google.com and Google.cn are exactly the same engine.  It doesn’t make any sense to search on Google.cn, where anything as innocent as 胡锦涛 (HuJintao) is obviously SEM manipulated. For the first experiment of the day we can see how, using this slightly conflictive term, results start to differ between G.com and G.cn. Try the links, see where there’s a Wikipedia article missing?

But the best of all is the answer given by the sample colleagues when I insist on why they use Google.cn: Oh well, the browsers here  direct you to Google.cn by default. That is probably the main reason why G.cn is ranked 3rd on Alexa for China, while G.com is only ranked 6th.

Hey, wait a second. Are you telling me that all it takes to get an identical, non SEM-ed Google Search in China is to type a “.com”, and 300 million netizens haven’t noticed in the last 4 years? Well, yeah. Kind of. Let me introduce you to:

The Chinese censorship and its peculiar victims

This is one of the most misunderstood aspects of Chinese censorship in the West. I realized this with the crazy Wang post, the one that was linked in an article 3 days straight on the Most Read list of the New York Times. I got lots of hits, and also lots of mail from creative Americans proposing ideas to help “free the Chinese” from the claws of the GFW.

But listen, the sad reality is, the CCP’s systems of censorship are so effective not because they are diabolically sophisticated, but because… because the Chinese netizens can’t give a damn if they are being censored by their government or not.

You don’t believe me? Then perhaps you have a better theory to explain why nobody uses the widely available, free web proxies to surf the internet. Or why the majority of Chinese netizens still use Google.cn when they have an identical search engine that is not manipulated on Google.com

Shocking, right? But not so much. The truth is that, in spite of popular funny memes and the occasional juvenile rant, the majority of Chinese who are rich enough to use the internet are happy with the status quo. They do find it mildly annoying to be treated like children by the CCP, but as long as the bills are paid, they don’t think so much of it.

And this is also why, if someone wants to create a device against the GFW, the user activated systems like proxies or Tor are not effective, because people simply don’t use them. The idea of a Server Side Proxy, or the Unblockable Host that would unblock a site WITHOUT action by the end user, was discussed here, and I concluded it was not feasible.

This is also the reason why initiatives like Chrter 08 never make it in China: it is not about users trying to get access to dissident sites, it is about dissidents unable to market their ideas to a general population that is unreceptive.

Advanced SEM for Dummies (Search Engine Manipulation)

The most amusing thing in the Google crisis is all the commentators crying about the loss of Google.cn and its negative consequences for the freedom of the Chinese. In fact, I maintain that Google.cn is the most evil product to ever have existed in the Chinese internet, and the World will be a better place without it.

That is because, unlike the Chinese official sites that practice censorship, what the search engines do is manipulation. Why? Because Google.cn is not a content site in itself, it is a gateway to the internet. When people type in a keyword into the search field, they are actually trusting it to return a fair picture of what is on the net.

When you type a “sensitive” term and G.cn removes all the results except the People’s Daily and Xinhua, Google’s responsibility is double: not only it supports those often objectible views on the first page, but it also implicitly states that it is the ONLY opinion existing in the World.

And the worse is, the Chinese who believed that would be right to do so, because Google’s well known principles clearly specify their commitment to give all the information available  in a democratic way. The little warning message that is displayed on Google.cn SEM searches is meant to avoid this situation, but it is tiny and often placed right at the bottom of the page, so most Chinese users just ignore it.

In the case of Google.cn, SEM is not about “good” or “evil”. It is about breaking the very principles that give a sense to the Google company, and it is understandable that Google has never been comfortable with it.

All tests in Chinese, English spelling is on purpose. The anomaly in the chinayouren string proves that in some rare cases G.cn does give better results that G.com, as SEM does not apply to petty disharmony.

Google.com and the GFW URL/IP blocks

One clarification is necessary here: when we say Google.com is free of SEM, it doesn’t mean that it is entirely uncensored. Google.com is hosted outside China, so it suffers from GFW blocks, in particular URL string blocks, like all the sites hosted out of the mainland. A funny way to test this block is to introduce the string  “chinayouren.com/eng” in any site search like Wikipedia, Yahoo, etc hosted out of China. The RC block is guaranteed! (fortunately I have changed my URL since and I  managed to unblock my blog).

A less funny example is when you search things like FLG, the famous religion fiercely opposed to the CCP. Google passes the characters of FLG to the URL, and this triggers the RC. And here is where we see the evil of Google.cn compared to Google.com:

• Google.com gives a very disruptive RC,  unmistakable sign of GFW block. This is plain censorship. Try here.
• Google.cn gives results from People’s Daily and Xinhua, the second of which starts: “FLG: the evil nature of the bloody…”. This is SEM manipulation. Try here.

Add to this that, for obvious practical reasons, the Chinese government cannot RC block too many common terms, or it would make the internet unusable. There are far less terms RC’ed on Google.com than terms SEM-ed on Google.cn, which is the reason why it is fairly easy, using Google.com in China, to get to very explicit material about Tiananmen 89 and many other delicate subjects. And even when those sites often are blocked themselves, the user who has seen the excerpts and thumbnails in the search results and then obtained an RC when clicking on them should be enlightened.

In conclusion, Google.com is a Search Engine that is:

1- Exactly as good quality as Google.cn (identical index)
2- Without the manipulation of Google.cn
3- AND much less censored than Google.cn

And the only reason why Chinese don’t use it is that Google.cn sounds more Chinese to them, and they just don’t care enough. Most probably the disappearance of G.cn will push the present G.cn users to switch to G.com, and the outcome will be increased freedom in the Chinese internet.

Conclusion

It is obvious that, if Google manages to keep Google.com and eliminate Google.cn, and in the process convince the CCP that it has already suffered the deserved punishment (for Google it DOES matter to lose G.cn, it is a source of revenue) then all will have ended well. That was probably not the initial intention of Google’s move, but it is a possible outcome.

On the other hand, some commenters are already saying that I am too optimistic, and that the CCP will quickly come to the same conclusion I have come and block Google.com. The good news is that EVEN if they do block Google.com, the situation may still be better than today. The Chinese Google users will start to miss the G, and they will start to use web proxies to access Google.com, expanding their use and making the Chinese net population more conscious of the GFW and of the ways to cross it.

And I think with this I have said all I had to say about Google and China. I am planning to write about other things for a while, at least until important, first hand news come out. In the meantime, you can ask questions or raise objections in the comments below, and I will try to help.

.

*NOTES AND CLARIFICATIONS:

1- If you are outside mainland China probably the experiments will not work, because Google.com is tailored to your own country. There are web proxies hosted in the mainland that you can use to trick Google into thinking you are in China. I don’t remember any now, if anyone knows please comment below.

2- People have asked me about Bing and Yahoo’s SEM policies. A very quick test shows that in principle Yahoo.cn/com is similar to G.cn /.com, while Bing only has one version here, and it looks SEMed, but I’m not sure. In any case those sites are way behind Google in China.

3- Many tech-savvy Chinese obviously DO know that Google.com is not SEM manipulated. And no, I am not implying here that most Chinese are stupid. I am just saying that most of them don’t give a damn if they search for Hu Jintao and the only answer they get is the People’s Daily bio. There is just no notion of the advantages of diversity of information, something that the CCP is not very good at promoting.

Prevention – Try to stay out of trouble

From the beginning when you set up your website, there are a series of measures you can take to reduce the probabilities of getting blocked and/or making your life easier if this happens. If you follow these points hopefully you will never need get to the next Section.

• Be careful with what you publish. >>>
• Try to avoid writing GFW keywords. >>>
• Choose where you want to be hosted. >>>
• Choose a good, flexible hosting service. >>>
• Host your blog/site on a subdomain. >>>

Action – When trouble is at your door

Then one day you realize that your Chinese readership has fallen to zero, and you wonder why you can’t open your website from China. If this happens to you, these are the simple steps to follow:

• Make sure it is really the GFW. >>>
• Check if there is an IP block. >>>
• Find out if the target is really you. >>>
• Check if there is an URL block. >>>
• Move to a new IP address. >>>
• Change your URL and Redirect. >>>
• Check that you don’t have links. >>>
• Try to eliminate the keywords. >>>
• Take it easy, and send feedback. >>>

Notes and Disclaimers

• Don’t forget to read the party of the first part >>>

.

PREVENTION

From the beginning when you set up your website, there are a series of measures you can take to reduce the probabilities of getting blocked and/or making your life easier if this happens. If you follow these points hopefully you will never need get to the next Section.

1. Be careful with what you publish. Sites in English have more leeway than Chinese to publish political content, but the more popular your posts get the tighter the line will be. Any kind of political activism can get you blocked, in particular subjects that are seen as potential menace to the one-party system. You can easily be misunderstood if you mention those subjects, even if your intention was different. Apart from politics, the other big subject that can get you blocked is porn.

2.Try to avoid writing GFW keywords. OK, if you need to write about these subjects, then try to avoid writing GFW keywords. To test which words are GFW keywords, run a search on Wikipedia in China, and see if it resets the connection. There are many ways to avoid keywords, such as periphrasis, or use of similar characters (omicron “ο” for “o”, etc.).

The problem is nobody will find your words on Google if you use this, so up to you to see how much you want to trade safety for search engine hits. Bear in mind that keywords won’t get you blocked automatically, and if your site is small enough you can relax and write as many as you want. The risk is some day a big website may link to you, then your keywords will attract the attention of GFW and your trouble starts.

3- Choose a good location for your host. If you are legit business with a business license you can apply for a permit and host your website in China. The permit is easy to get, as long as you don’t do politics or journalism. This solution will make your site faster for users in the mainland, and it will avoid all sorts of trouble with the Great Firewall.

If you are planning to write some controversial content, or if a large part of your readers will be outside the mainland, then it is better to host out of China, otherwise you risk to get your whole site closed down and your data lost. A good location in the middle is Hong Kong. It is well connected internationally and close to one of the nodes entering the mainland. Therefore it should be quick on both sides, plus many Hong Kong hosts are reportedly understanding with the GFW problems and they will help you to get rid of an IP block.

4- Choose a good, flexible host. Regardless of where you are hosted, the most important is that you choose a hosting company that will give you flexibility and will try to help you if anything happens. In particular you should check if there is an affordable option to change your IP. In my case it was 30$/year, I heard from other webmasters they paid double. The essential is to check that your host will give you this options, because not all hosts offer it.

5- Host your website/blog on a subdomain. I learnt this trick in my recent experience. When you get an URL string block, it will not necessarily affect the whole domain, but only the particular subdomain where your site lives.

If you host your site on a subdirectory, such as www.domain.com/blog/, this will allow you to deal with a URL string block very easily, just by moving to, for example, www.domain.com/myblog/. This way you can save the money and the trouble to register a new domain, and you can continue to use that domain that your users already identify with you.

.

ACTION
Then one day you realize that your Chinese readership has fallen to zero, and you wonder why you can’t open your website from China. If this happens to you, these are the simple steps to follow:

1. Make sure you are actually blocked by the GFW. The first obvious step is to check that you are really blocked by the GFW, and not by some other problem. You should get in contact with some friend who is outside the mainland and a friend inside the mainland. If it is a GFW block, the outside friend should see no difference at all, whereas the inside one should get a Reset Connection error or sometimes a Server Time Out error when he tries to open your site. There are also online tools available to do the test, but  I haven’t tried their accuracy.

2- Check if there is an IP block. The first thing to do in this case is to check if you have an IP block. It is very simple, from China type your site’s IP in the address bar of your browser after the “http://” and if it doesn’t work  then it is an IP block. IP blocks are the most common GFWs blocks so this test should normally confirm step 1. NOTE: If you don’t know what is your site’s IP there are many Firefox free add-ons, and other web based services that will give you this information.

3- Find out if the target is you, or some site living on your server. Very often when you are down to point 3 what is actually happening is that you are collateral damage of the GFW. They are not targeting you, but just somebody else’s site which lives in the same IP or the same subnet (group of IPs). It is not easy to check this, but there are 2 simple techniques that can give you a clue. One is to use a reverse IP service, such as this one, where you can see more or less how many sites are sharing IP with you, and you can get a list of these sites. If you are hosted in the same address as FLG.com (ie. an anti-CCP organization) then you know it was not you.

4- Check if there is an URL block. In China, go to Wikipedia search box (or some other search bar on a site hosted out of China ) and type the strings you want to test. First try domain.com, and then domain.com/blog. If you are lucky only the second one will reset your connection, which means that your domain root URL is still unblocked and you can easily switch to a new subdirectory as explained above.   NOTE: URL blocks typically give RCs. When you do RC tests you must wait for at least a minute after getting a Reset Connection, otherwise you will RC again no matter what you type in (the connection is still not re-established)

5- Change your IP address. OK, so you have done all the checks. It is time for action. Supposing you followed the prevention steps in the first place, you have a cool, understanding host that is not pissed off each time you get his whole subnets blocked in China, and who is willing to swap IPs for a moderate price. Get in contact with him or select a new IP directly from your admin panels functions (if available) from the site.

6- Change your URL and Redirect. This part can be very easy or very difficult, depending largely on how well organized is your website. In any case, the process is not without pain, and you will lose many incoming links. Why? Because even if you have a Redirect on your old URL, anybody in China clicking external links to your old site will get an error, as the connection resets before your server has the time to redirect to the new URL.

If your blog is powered by a good system like WordPress, and if you have been consistent with the use of permalinks over the years, it should be very easy for you to move to the new directory in your same domain. Just follow the steps given in your blog engine and you will be done on a few minutes. If you need to move to a new domain because the whole domain URL is blocked, then it will take a bit longer, but it shouldn’t be a big deal.

7- Check you don’t have remaining requests to your old URL. Many of the blog’s elements, such as pictures, are probably still linked to a URL in your old subdirectory, and therefore when the page loads they instruct the users computer to connect to that subdirectory in order to display the picture. If you have been paying attention you know what will happen in this case: the fretful RC. Don’t panic, run a search in the whole blog (not the search for users on the website, the admin tool in your blog admin panel, you might need to install it as a plug-in. In wordpress it is called Regex Search) and replace all instances of the blocked URL with the new URL.

At this point, you might still see that the blog continues to trip RC. In this case it might be some JavaScript or other active content in your site that is sending requests to the old URL. Use a traffic viewing tool like the free Firefox add-on Tamper Data which will easily show you all the requests sent while loading your site. Type the blocked string in the search bar and you should see the old URL coming up at some point, and the software will tell you which application is the culprit. There is a bit of detective work to do in this case, as it is not always obvious which widget or active element made the call.

8- Try to eliminate all keywords. If you have followed the instructions up to now your site should be open again in China. What I recommend you to do is to run a search for keywords in your blog and try to eliminate/disguise some of them. Especially look at the posts that were linked by big websites and they received a lot of hits in China. NOTE: you don’t need to eliminate outgoing links, a link to a forbidden site does not usually get you blocked, otherwise every major website in China would be down.

9- Take it easy, and tell me about it. Go back to step 1 “prevention” above and try to get smarter so next time you don’t have to go again through the whole process. Or else remain blocked and take it as a honour, make yourself a T-shirt saying “I was blocked by da GFW”, after all not every site is worth the effort of blocking. You must have had at least some relevance in the eyes of the GFW for it to censor you. In any case, if you get GFW’ed and follow these steps, give me your feedback in comments below so we can improve the method.

.

NOTES AND DISCLAIMERS

• There is no such a thing as censorship in China, the GFW is due to a bug in the Chinese internet that defies all scientific explanation.
• These instructions are empirical knowledge, I have no internal information on the GFW, and the basis for these instructions is my own experience and that of a few other bloggers shared with me by email. I am not giving credit to anyone here to avoid getting them in trouble again, but guys let me know if you want to be linked.
• It cannot be excluded that the GFW reads these instructions and changes its blocking devices. Also the GFW has a random nature and not all the steps above will work in al the situations.
• These instructions are best used by websites of a small to medium size who are blocked more or less automatically by the GFW. I you are big, especially if you have content in Chinese, then chances are you are followed by human censors. The IP/URL swap will not fool them very long, and they can take you down again easily if that’s what they want.
• If you are hosted within mainland China, GFW instructions do not not apply to you, instead you get what is usually known as the Net Nanny. See this post if you want to understand the difference.
• As far as I know there is nothing illegal in writing instructions for webmasters to deal with an internet bug. However, if this text is somehow illegal or inharmonious, I would be grateful to the concerned department if they can inform me so I can correct it before they take more drastic measures.

As time passed, we grew so used to all the fox capabilities that it became normal for an internet browser to perform the most various functions: Firefox was my Chinese dictionary, my anti-GFW proxy, my image editor, my wikipedia link, my financial consultant, my bookmark and my fluffy nail brush. It was Jesus, and it was perfect.

And then, suddenly,the fox got old. The add-ons started to weight on its worn out bones, and one day we found ourselves waiting 10 seconds for the browser to open. Never again. But what could we do, we were addicted to the add-ons, and way too proud to fall back on Explorer.

For months (years) on end there was no solution forthcoming, and no amount of reinstalling helped to solve the issues. Then the shrewd guys at Google jumped at the chance. They launched the new Chrome, a browser that actually does nothing at all except – guess what – browse at the speed of light. The first day I downloaded it “just to give it a try”, and after that test flight I never opened the Fox again. Instead, I got myself a dictionary.

So now the Fox is back. Its 3.5 iteration feels certainly much faster than the previous, and it supports, if not all, at least the most important of my add-ons. The influence of Chrome is very obvious, with new functions like the “private” surfing, one-click bookmarking and that little button on the right hand side of the tabs that comes so handy to open a new one.

As for the speed, I seriously doubt it can beat Chrome (look at the announcement, Mozilla compares it with Explorer, but avoids mentioning Google). But as long as it is reasonably fast and it continues to brush my nails, I think it deserves another chance. Let’s see how it goes.

I am back with the fox. I hope it rocks.

We will be looking at Google.cn, Google.com and Baidu.com, and we will try in each of them 3 different kind of search terms.

A- Chrter 08: In all its combinations, which are 08宪章 and 零八宪章
B- Political Terms: Tiananmen incidents (天安门六四事件), FLG.
C- Vulgar words: Sex. I will employ the “blog job” and the “chicken bar”.

It is understood that in all cases the search terms are in Simplified Chinese. The browser is Firefox 3.0.5. and the connection is a normal home DSL by China Telecom. The possible results are:

• Free Search – Results look consistent and realistic, like the ones obtained in the West.
• Reset Connection (RC) – This can only be seen in Mainland China. The result is an image like the one below and the search engine cannot open anymore for a while (I estimate 30 seconds). RC is not directly done by the Search Engine. Wikipedia internal search also gives RCs for B Terms.
• Forbidden Message (FM)  – This is the forbidden Message that, with slight variations, is the same as shown below. It says something in the lines of: “Some results are not displayed according to the local laws, regulations and policies”.
• Manipulated Results (MR)- This is the case where the results are obviously manipulated, for example in the search of 天安门六四事件 (Tiananmen incident) on Baidu, where all the results are official newspapers such as People’s Daily, etc. Sometimes it can also carry on top of the page a FM.

Google.com
A -Free Search.   (But click some individual results gives RC).
B- Reset Connection
C- Manipulated Results.

Google.cn
A- Forbidden Message and (sometimes *) Manipulated Results
B- Reset Connection.
C- Forbidden Message. When used “” gives Manipulated Results.

Baidu.com
A- Manipulated Results. When used “” gives Forbidden Message.
B- FM and Manipulated results.
C-FM and Manipulated Results.

Conclusions

1- The results are somewhat erratic and it is difficult to see a pattern: it all looks like a series of patches on top of each other rather than a systematic implementation. Also, things change in time, as in *, where the Manipulated Result I saw Sunday cannot be seen anymore.

2- Baidu has a different system from Google: it has no Reset Connections. This is very advantageous for Baidu and I understand it is unfair competition, as a RC is one of the worst experiences while surfing.

3- This might be due to Google’s own preference server location: the involvement of the Search Engines in the RC is unclear no direct involvement (even Wikipedia has RCs!) whereas Manipulated Results obviously requires their action, and can more easily attract attention from Advocacy Groups. Of course, in the case of sexual terms (C), this is not a problem as the Manipulated Results can just be called “Safe Search”.

4- The Chrter 08 has different treatment than other political terms, but it might just be because it was banned urgently and suddenly, so it is only a quick fix added to existing structure. It does not provoke RC in any case. It looks like they have decided to leave it alone on Google.com to avoid attention from Western advocacy groups, but in exchange Google has had to give up Google.cn and apply the infamous “porn block” to it which is active censorship by SE. Why the FM and not RC? Who knows, I am guessing perhaps RC is more complicated to implement.

5- In any case, and however negative, I understand it is always better to show FM than Manipulated Results, because the former is openly admitting censorship, whereas the latter is a lie and a distortion of reality. Forbidden Message does increase transparency, yet does not justify involvement in political censorship. From this perspective, Google is closer to the truth than Baidu. Baidu seems indeed a more active participant in the government’s information control schemes, and Chinese users of Baidu are clearly the most exposed to Search Engine brainwash.

UPDATE: Following corrections by international expert Nart Villeneuve below: I have introduced a few changes of my own (in blue). In any case, this post is just a very basic review of the SE Censorship system from the perspective of a normal user. If you really want to understand how the GFW works, you should read proper research papers like this one, or this one.

.

IMAGES:

1- FORBIDDEN MESSAGE (FM)

2- RESET CONNECTION (RC)

NOTE: If someone is interested in this or has some more information to share please put it in comments. Unfortunately my time is very limited so I only ran 2 or 3 terms for each of the classes A, B and C above. There might be things I overlooked and I would be grateful if you can point them out.