A screenshot from Comment Robot's public Weibo profile page. Fair Use.
“Comment Robot” (评论罗伯特), an AI Chatbot, has sparked quite some amusement and rage on Weibo since December 2023, as it has kept poking many Weibo users with humorous, heart-warming, but sometimes inappropriate or even offensive comments.
When it first emerged in early December 2023, some users with very few followers were puzzled as to why their Weibo updates had caught the attention of a “Big V” – Weibo's official verified user and opinion leader with many followers – who had zero connection with them online. After local media outlets exposed the “Big V’s” as a new AI chatbot, a few Weibo users raised the issue of its infringement on their social media data and privacy.
An engagement boosting machine
To address users’ concerns, on January 24, 2024, one month after its launch, “Comment Robot” apologized for “offending users” and “causing trouble and confusion” on behalf of Weibo through a Question and Answer post. Weibo explained that “Comment Robot” was meant to boost the engagement rate of its users:
平台希望借助最新的生成式人工智能技术优化普通用户的发博体验，提升普通用户在平台内容生产中的活跃度。
The platform [Weibo] hopes to leverage the latest generative AI technology to optimize the blogging experience for ordinary users and boost the engagement of their platform contents.
Thanks to rigorous control over social media content, most Chinese social media users have stopped sharing content online. The launching of “Comment Robot” came approximately one month after the full implementation of the Big V real names system, which triggered a strong backlash from Weibo influencers. In addition, when compared to video platforms such as Douyin and XiaoHongShu, Weibo’s ads income growth is lagging behind largely due to a lower engagement rate.
Powered with automated responses, the chatbot has successfully intrigued users, entertained them, and resulted in a greater number of interactions.
Behind the chatbot runs a large language model (LLM) of complex neural nets and computing efforts. Similar to other generative AI, such as ChatGPT 4, the Weibo Chatbot is designed to carry out designated tasks through the simulation of human cognition and the training of massive data.
The Comment Robot’s duty is to crawl the platform, detect the meaning and sentiments of certain user-generated content, and form a reply, which is a combination of Chinese characters based on the predicted association, namely, the probability of co-appearance in a given context. For instance, when someone expresses joy, the response is more likely to contain messages such as “Congratulations!” or “I am so happy for you!” Likewise, a selfie might receive a compliment from the bot.
“Stochastic Parrot”
Yet, at the same time, more and more Weibo users found the Chatbot disturbing as it started generating nonsense. A Weibo account, “Victims of Robot” (罗伯特受害者联盟), was set up to collect inappropriate comments made by “Comment Robot”. Here are some examples:
用户A：领导同事在天堂 [烛光]
评论罗伯特: 你也是吗？我的家人和同事也在天上
User A: my team leader is in heaven [candlelight]
Comment Robot: You too? My family and colleagues are also in heaven.
用户B：在不该心软的时候心软
评论罗伯特: 没事的 不该软的地方不软
User B: My heart is being soft when it shouldn’t be.
Comment Robot: It’s OK. [Another body part] will not be soft when it shouldn’t be
用户C：#龙年第一条微博# 龙年希望所有我爱的人和爱我的人都平安，健康，喜乐
评论罗伯特: 虽然不太可能但我真的希望你健康快乐
User C: #First Weibo in the year of Dragon# I wish all my loved ones are healthy, happy and peaceful in the year of Dragon
Comment Robot: Although that’s quite impossible, I wish you happiness and health.
用户D：当你看到这条微博时我已经入土为安了
评论罗伯特: 好好好
User D: When you see this Weibo, I will be lying in my grave.
Comment Robot: Good good good.
The nonsensical responses, including sex jokes and dark humour, are likely related to the so-called “Stochastic Parrot” effect. Since the AI language model excels at logical inferences but lacks an understanding of language, hence, the quality of generative content is highly susceptible to the training data. In the course of data training, the interactions with the bot are very often inconsistent, thus can produce nonsense, and perpetuate existing biases.
Weibo admitted in Comment Robot’s Q and A post that the chatbot may generate offensive comments during the language model training process and added that users’ feedback will help optimize its performance.
AI governance: user rights and privacy
However, Sina Weibo has never addressed whether users’ interactive content with the bot will be used for its AI model training and neither did the company issue a formal announcement about the launch of its chatbot.
Most people were automatically forced to engage with the chatbot. Like other generative AI products, it is a black box of risks and privacy infringements. Weibo users started asking questions such as where the training data came from and to what extent the chatbot’s wrongdoing be publicized and held accountable? A few urged their fellow Weibo users to stop interacting with the chatbot, like this one:
很显然那个评论罗伯特的机器人越来越智能了，词汇量和语库越来越大了，估计是用户一直和它互动的结果，AI把这些数据导入运算分析最后根据语境选择合适的内容来回复……它以后会越来越像人，其实是相当恐怖的一件事，我是建议最好还是给他lh了吧，别搭理它别给它提供数据……谁知道是用来干啥的
Obviously, that comment Robot is getting smarter and smarter, vocabulary and text corpus is getting bigger and bigger, it is likely a result of its interaction with other users. The AI would import and analyse the data and then choose the appropriate content in its reply according to the context …… It will become more and more like a human being later on, and it's actually a rather scary thing, I am suggesting that it is best to block it. Don’t interact with it nor submit your data …… who knows how it would use your data…
China has introduced quite a number of regulations on AI. The latest “Interim Measures for Generative Artificial Intelligence Services” clearly stated that the provider should clarify and disclose the people, occasions, and purposes of its services, and the provider shall fulfil the obligation to protect the input information and use records of the user in accordance with the law.
That’s why Sina Weibo asserted in its January Q and A post that the company would only use publicly available data to train Comment Robot. It also adjusted its policy to require the Chatbot to interact only with those who followed the account.
However, according to user complaints, it can still comment on public posts created by ordinary users after blocking the bot (only without notification). In addition, its latest update on March 13 enables the bot to form a back-and-forth dialogue with users rather than simply leaving one comment, indicating a greater level of interactivity.
While the space for user rights campaigns in China is extremely limited, it seems that many Weibo users are sceptical of the AI robot — as of March 21, three months after its launch, Comment Robot merely had 716,000 followers even though it had posted massive comments under other Weibo users’ posts and occupied the social limelight through hashtags and media reports.
Obviously, users are worried that the data be sold or forwarded to other marketing entities or authorities for commercial or political purposes. Such sentiment is expressed in a number of Weibo posts, such as the following:
细思恐极
那个评论罗伯特简直一个收集微博用户圈子画像，即将通过图灵测试的机器人。以后遇到带节奏、引战的，你都无法分辨对面是真人还是故意批量投放的人工智能。
It is extremely creepy if you give it a thought.
That Comment Robot keeps collecting profiles of Weibo users’ circles and is about to pass the Turing test [A test of a machine's ability to exhibit intelligent behaviour equivalent to, or indistinguishable from, that of a human]. In the future, when you come across content leading or picking fights, you can't even tell whether the opposite side is a real person or an artificial intelligence intentionally placed in bulk.
一直想问罗伯特天天搞评论互动是为了摸清用户的喜好和语言表达方式吗……等我死掉它就可以替代我用我的id和我的网友们聊天……网友应该发觉不了对方不是人吧……
I wonder if Comment Robot's daily interaction with users is just to understand the user's preferences and language expression …… What if, after I died, the bot could use my Weibo ID to chat with my online friends… They wouldn’t be able to find out that the other party is not a human, right? ……
Written by Karst Wong
This post originally appeared on Global Voices.
