Over the past two years, China has seen a proliferation of state-endorsed experiments representing early stages towards a national social credit system. These pilots evaluate citizens based on opaque formulae reportedly incorporating financial, social, behavioral, and legal data; and reward or penalize them by, for example, facilitating or restricting travel, credit, and shopping. (China Law Translate is currently translating a series of related government documents.) The final system, officials hope, will “allow the trustworthy to roam everywhere under heaven while making it hard for the discredited to take a single step.”
Such resounding language, together with suspicions about political metrics and the schemes’ eye-catchingly dystopian long-term potential, have tended to overshadow the pilots’ relatively limited, piecemeal, and provisional current state. But the nascent social credit system is indeed part of “a broader Chinese government push to harness big data as a resource for social control,” according to Shazeda Ahmed, a UC Berkeley PhD student and former fellow at the University of Toronto’s Citizen Lab. Ahmed has previously written about her investigations of the social credit pilots and related issues at Citizen Lab, and discussed them this week with CDT:
CDT: Who is conducting these pilots? How and when did they start? What are the goals?
Shazeda Ahmed: Right now there are two types of pilot testing, private and public. The Chinese government has given eight companies official permission to do experimental testing of social credit systems, and the one that’s publicly available is Sesame Credit, owned by Alibaba subsidiary Ant Financial. The public social credit pilots are being conducted at a city level in over thirty cities around the country, with a prominent example being Shanghai.
I think the testing started sometime in late 2014 or 2015, when the first drafts of regulations suggesting how such a system could work were issued and when the first Western media articles critiquing the system were published. The purported goals of what the state is hoping will one day be a nationwide social credit system differ depending on what you read. I’ve seen government officials cast social credit as a way to reinstate trust in people and institutions. Both individuals and businesses get social credit scores, and there is an emphasis on the notion that data produced from their transactions can be used to accurately monitor how reliably they adhere to laws and social norms. I’ve even seen a few Chinese editorials where people make the complete opposite argument that’s common to Western condemnatory takes on social credit, and insist that social credit would actually protect civil liberties. This is related to one of the other alleged goals of the system, which is to issue credit to traditionally “unbanked” populations– migrant workers, college students, rural citizens who all seek to take out loans. Although they may stand to gain opportunities through social credit schemes, they are also highly vulnerable and could end up in debt, among other consequences.
CDT: But they wouldn’t have to resort to nude selfies to get loans?
SA: I’ve read about this type of loan collateral before and can’t say I know much about it, but yes, that is definitely another way that less well-off and potentially less tech-savvy individuals who need to borrow money are exploited. I remember the first time I read about it and hoped it wasn’t true! To me, making loan disbursement conditional on something that can be used to blackmail people seems to defy some of the (admittedly vague) concepts of morality that are loosely articulated in the social credit system. I sincerely hope that at the intersection of still-developing privacy protections and regulations about “indecent” content that can’t be posted online in China, protections against this sort of harassment emerge.
CDT: How do the various pilot schemes differ?
SA: At present I don’t have a clear sense of how the private company-owned social credit programs differ from the municipal government-run ones, although it seems that through the former users of financial technology or “fintech” apps can access their scores instantly, whereas with the city-level social credit systems one might need to show up at a specific office to find out one’s credit score. I have a lot of unanswered questions about the variance between these separate systems, how they may or may not converge if the goal is to create a unified system, and whether or not this is intentional on the state’s part to have a variety of results to compare after the pilot testing ends.
CDT: What kind of data feeds into the system?
SA: Taking the concrete example of Sesame Credit, within the app there’s a visual that breaks the credit score down into five components: users’ credit history, behavioral habits, ability to pay off debts, personal information, and social networks. Recently I’ve seen articles that suggest data including criminal and other government-held records will also be included. The actual algorithms that compute credit scores are black boxed, which means that it would be illegal to do experimental testing to figure out which transactions and behaviors would cause a score to rise or fall. When I wrote about this for the Citizen Lab, I sought out statements from Alipay and Sesame Credit company representatives about which inputs affect scores, and found many peculiar and highly specific factors including hours of video game play (score declines) or purchasing diapers (assumption that user is a parent, thus score increases). These statements seemed to suggest that contrary to the language of objectivity the state uses to describe this system, social credit is (surprise) based on subjective Chinese social norms and practices.
CDT: Without incriminating yourself, have you been able to do much independent exploration of how the scores are calculated?
SA: All of the research I have done on this so far is based on openly available sources such as Chinese news media, Sina Weibo, IT industry journals, and web forums. I can’t say I’ve seen many fully fleshed-out theories of how the system works, though I have come across more than a few Chinese articles and threads suggesting that the way to raise one’s Sesame Credit score is to do more shopping using Alipay. This reinforces my own hunch that the story in the shadows of the Western media portrayals of “social credit as Big Brother” is actually one of social credit being about spurring consumerism in a country where people have historically saved more than they’ve spent. In a way this is a banal point about how social credit encourages capitalism in China, because of course this is the point of credit scoring. Yet it’s still worth raising because I think social credit will be inextricably linked with fintech, and it’s important to consider how mechanisms of reward and punishment that are tied to one’s personal finances and behavior (online and offline) will develop in light of this bundling.
CDT: What are some of the current consequences of high or low scores?
SA: Since I last checked, high Sesame Credit scores open up access to expedited security checks at the Beijing domestic airport, as well as a faster visa application process to places like Singapore and Luxembourg (with other countries having expressed interest in this as well.) Speaking to my point about consumerism above, it’s not particularly shocking that the countries that currently are or plan to partner with this express visa process are also ones where Alipay users can pay for products using the app. So to reiterate, we can imagine a scenario in which someone has a high credit score that gets them a visa to one of these countries relatively quickly, and once they’re in said foreign country they can open up the Alipay app to find out about local sights and shops, as well as make payments using Alipay (and potentially rack up a higher credit score in the process). Other benefits include having online purchases delivered to one’s home to try out before paying for them, and renting cars or booking hotels without putting down a deposit. Punishments are less clear at the moment, but primarily include blacklisting and fines. An example I’ve seen mentioned in a few places is the Supreme People’s Court’s list of people in extreme debt, which is apparently shared with Alipay so that those people cannot make “luxury purchases” (undefined) in the app or board flights or high-speed trains. I’m not clear on how this would affect their credit score, but it would appear to fall under the Sesame Credit category about fulfilling commitments and obligations.
CDT: Some comments from Ant Financial suggest that the system is not just about passive tracking, but active “social sculpting.” Beyond encouraging consumption in the way you’ve discussed, this seems to include incentivizing activities like charitable donations, and discouraging others like late night web browsing and excessive video gaming. In your Citizen Lab piece, you even mentioned “the possibility of social credit and mobile finance access being blocked to penalize citizens for acts of protest.”
SA: Yes, while I have no hard evidence of this sort of blocking having yet occurred, it’s certainly not difficult to imagine how easy it would be to instate. In their article about social credit The Economist raised this point as well. There is definitely a drive to change people’s behavior, and one of the questions I’m hoping to explore through interviews with social credit users is whether they are behaving differently in their daily lives to try and raise their scores. After all, it’s not yet a nationwide system, the benefits may not appeal to everyone, and the punishments seem to thus far be quite minor. This also raises the question of if hackers have figured out how to artificially raise their scores and are riding the benefits of that. China’s not known for designing hacker-proof tech, and Alipay in particular could do with better cybersecurity practices. Pretty recently a friend told me about how Alipay had a feature where if you forgot your password, you could identify a few of your friends and recent purchases to reset it, whether on your phone or someone else’s. [Facebook uses similar verification features.] Naturally people took advantage of this, and Alipay’s quick fix was to keep this feature but only limit it to the phone the user had registered with the app. Similar events are bound to occur, and lax security precautions make me wonder if in the future people will be able to purchase pre-curated accounts that already have high scores built into them.
CDT: How far beyond the kinds of tracking commonplace in the West do the Chinese systems look set to go?
SA: I think that depends on how well-integrated the private company-run systems become with government bureaus. While the municipal-level social credit systems are worth paying attention to, I think the range of data points a source like Sesame Credit has access to is far greater and can be put to any number of uses that the company and the state may not have even realized or articulated yet. It’s also imperative to pay attention to what gets tracked through fintech and social credit apps when users are abroad, as I mentioned above. The power to monitor citizens while they travel shouldn’t be underestimated. Finally, I’m really interested in seeing how this kind of system might serve as a model to developing countries that see China as an exemplar of social control. If digital citizenship initiatives were to become more popular along with social credit scores proliferating in other countries, it wouldn’t surprise me if scores were someday used to make decisions about immigration and international travel, for example.
CDT: Western coverage of the social credit system has not always been entirely sober. How have observers been most carried away?
SA: I have to admit that I’m growing tired of seeing social credit compared to 1984, especially because it’s not been rigorously researched yet and is bound to change in ways we can’t predict. There are several comparisons to the Black Mirror episode “Nosedive” that are funny but make the assumption that somehow everyone is already receiving a social credit score and that they’ve structured their lives around raising it.
CDT: What do we know so far about how users feel about the schemes, and how seriously they take them?
SA: The few interviews I’ve seen with Chinese citizens who are using it suggest that they’re either reserving judgment or not adversely affected. This too can change, and while I think drawing attention to social credit is incredibly important, I’d like to see more impressions from average users to get a sense of how it will spread and be received across China.
I’ve poked around some Chinese forums where techies congregate to talk about how to raise scores, with a notable example being Zhihu (kind of like Quora). People there seem to think of social credit as being more of a game, and have made claims about raising scores by making specific purchases. While these are definitely interesting takes on the system, it’s hard to know how seriously to take them and how accurate these findings may be.
CDT: Is there a risk that, despite claims that the schemes will extend credit to people who haven’t had it before, they might end up exacerbating existing inequalities? That you’ll need to have social credit to accumulate social credit?
SA: This is another concern I’ve seen raised in web forums, and I’m not sure how that’s played out so far. It’d be great to gather some empirical evidence of this, because it seems highly possible that the system is cyclical as you’re suggesting. There has been minimal discussion of how to address such an issue, or the related problem of contesting a score change that someone feels is an inaccurate reflection of their behavior.
CDT: What are the other privacy implications? What kind of protections are in place for all this pooled data?
SA: One that comes to mind is that there is inadequate transparency around which inputs determine a credit score, which could mean that users are being judged based on factors they did not consent to sharing for this specific purpose. I think the greater privacy infringements right now can be found in the local investigatory services through which one can easily buy Chinese citizens’ data, which David Bandurski and the Southern Metropolis Daily recently wrote about in depth. There are some privacy provisions under the new cybersecurity law that will go into effect in June, but there appears to be far greater concern with keeping data on Chinese citizens stored in China than there is for who within China has access to that data. To me the biggest concern is about who the data-collecting companies share citizens’ personal information with– third parties? The state? It’s still unclear.
