Transcription

REPORTConsumer-friendlyscoring

Recommended citation:SVRV (2018) Consumer-friendly scoring. Gutachten des Sachverständigenrats für Verbraucherfragen.Berlin: Sachverständigenrat für Verbraucherfragen.Berlin, October 2018Report of the Advisory Council for Consumer AffairsISSN 2510 0084Published by:Advisory Council for Consumer Affairsat the Federal Ministry of Justice and Consumer ProtectionMohrenstraße 3710117 BerlinTelephone: 49 (0) 30 18 580 0Fax: 49 (0) 30 18 580 9525e mail: [email protected] verbraucherfragen.deWebsite: http://www.svr-verbraucherfragen.de/en/Design: Atelier Hauer Dörfler GmbH, BerlinPrinted by: Druck- und Verlagshaus Zarbock GmbH & Co. KG, Frankfurt am Main SVRV 2018

REPORTConsumer-friendlyscoring

2 AcknowledgementsPart of the mission of the Advisory Council for Consumer Affairs(SVRV) involves incorporating new research findings and practicalexperience into the drafting of its publications. In order to meet thisrequirement as comprehensively as possible, the SVRV has createdvarious serial publications – reports, working papers and commissioned studies. It also stages specialised events and engages in publicand non-public conversations with representatives of the academicand business communities and of civil society. With the aid of representative and non-representative surveys, public opinion informs thework of the SVRV, as do the legitimate interests of business enterprises. Without the assistance and cooperation of these individuals andinstitutions it would not have been possible to compile this report.The SVRV thanks all of the staff of its officefor their outstanding workon the preparation of this report. We extend a special word of thanksto the research staff – Johannes Gerberding, Dr Christian Gross, DrAriane Keitel and Sarah Sommer – as well as to Thomas Fischer, headof the SVRV Office, and to the temporary deputy head of the SVRVOffice, Stefan Kubat.Dr Felix Rebitschek, Jasmin Ghalib und Juri Ritz assisted the leadauthors of the report, Gerd Gigerenzer and Gert G. Wagner, in theirwork. Our thanks go to them too.This report is also based in part on material that has been publishedin the SVRV Working Papers series.The SVRV thanks the authors of the working papers Verbraucher-Scor ing aus Sicht des Datenschutzrechts (‘Consumer scoring in the light ofdata protection law’) and Dokumentation einer empirischen Pilot-Studiezum Wissen über und zur Bewertung von Verbraucher-Scoring (‘Documentation of an empirical pilot study on awareness and assessmentof consumer scoring’).We also thank the authors of the study Technische und rechtlicheBetrachtungen algorithmischer Entscheidungsverfahren (‘Technical andlegal reflections on algorithmic decision-making processes’) – Professor Georg Borges, Dr Matthias Grabmair, Daniel Krupka, ProfessorBurkhard Schäfer, Professor Erich Schweighofer, Professor ChristophSorge and Bernhard Waltl of the Specialist Group on Legal Informatics of the German Informatics Society.Before and during the preparation of this report, members of theSVRV and of its Bureau held numerous conversations with representatives of the academic and business communities and of NGOs. Ourthanks go to the following (in alphabetical order):

3 Professor Andreas Beyer (University of Cologne), the Bundesanstaltfür Finanzdienstleistungsaufsicht (Federal Financial SupervisoryAuthority), the Bundesversicherungsamt (Federal Insurance Office),Wolfie Christl (Cracked Labs – Institut für kritische digitale Kultur),Dacadoo AG, Generali Deutschland AG, Professor Justus Haucap(University of Düsseldorf), Lorena Jaume-Palasi (Algorithm Watch),Professor Genia Kostka (Free University of Berlin), infoscore Consumer Data GmbH, Professor Sergio Lucia (Technical University ofBerlin), Professor Stefan Lessmann (Humboldt University, Berlin),Professor Mario Martini (German University of Administrative Sciences in Speyer), Professor Steffen Mau (Humboldt University, Berlin),Professor Klaus-Robert Müller (Technical University of Berlin), WalterPalmetshofer (Open Knowledge Foundation e.V.), Professor Eberhard Sandschneider (Free University of Berlin), Schufa Holding AG,Professor Wolfgang Schulz (Hans Bredow Institute and Alexander vonHumboldt Institute for Internet and Society), Arne Semsrott (OpenKnowledge Foundation e.V.) and Sparkassen DirektVersicherung AG.A total of 75 business enterprises or health insurance funds took part ina market study covering the segments of credit, driver and health scoring. The SVRV offers its sincere thanks to them for their cooperationas well as to those who took part in a representative survey on publicawareness and acceptance of scoring. The survey was conducted byinfas, the Institute for Applied Social Sciences, with Ms Janina Belz ashead of project. We are grateful to all of these important partners.Dr Philipp Hacker of the Berlin Social Science Research Centre(WZB) and Christin Schäfer of the consultant firm acs plus: data withcare have earned our gratitude not only for their critical appraisal ofthe draft report in the form of an independent peer review but alsofor many important suggestions that have been incorporated intothe final version.Please note that this text represents a translation of the original report published in German, excluding the annex of the original report.Therefore, any reference to the annex shown in this text refers to theannex of the original report. The language in the text of this reportis intended, in principle, to be gender-neutral. For the sake of betterreadability, however, we have refrained from continuously referringto both sexes separately.Lucia ReischGerd GigerenzerGert G. Wagner

4 Executive Summary – Recommended actions for consumer- friendly scoring1. Making scoringcomprehen sible forconsumers1. The Advisory Council for Consumer Affairs recommends that data protection authorities operationalise the comprehensibility requirements set out in theGDPR (cf. Article 15 para. 1 letter h) for scoring andscore-based business processes. Comprehensibilityshould be measured according to the standardsrelevant to the average consumer. Where scoringentails a level of complexity that is no longer comprehensible to the individual consumer, measuresshould be taken to ensure that scoring processescan be understood not only by supervisory authorities, but, at the very least, by consumer bodies andnon-state actors as well.2. Scoring services should release clear and comprehensible information for consumers aboutthe main criteria used to score them and, preferably, how these variables are weighted. Tradesecrets, of course, must remain inviolable. Thedefinition of which variables are considered crucial for consumers cannot be left exclusively tolawmakers: this task should additionally fall within the remit of consumer organisations, or, alternatively, the “market watchdogs” of Germany’sconsumer advice centres. At any rate, full disclosure to supervisory authorities of scoring systems and their attributes is a must (see page 5 ofthe Advisory Council’s Digital Sovereignty report).Some members of the Advisory Council advocatefurther-reaching transparency. They believe thatall scoring variables should be disclosed to theconsumer and that the relative weighting of eachcomponent should be indicated in the calculationof the score. To this extent, any interests on the partof scoring services and users in maintaining secrecywould take second place to the consumer’s interestin receiving information. At the same time, the tradesecret of how a scoring system has been developedand programmed would be maintained.3. However, disclosure alone will not necessarily giveconsumers a better understanding of how scoringworks. This will require a variety of measures, whichinclude: providing examples of consumer scoresand how they are tiered according to different vari ables; the production of visual teaching aids (e. g.by consumer organisations); general efforts to raisescoring-related competence among consumers. Anyassessments of how comprehensible scores are toconsumers should be based not only on expertopinion but on empirical evidence.4. Consumers already have a right to tailored andmeaningful written information whenever they arescored (see Article 13 para. 2 letter f, 15 para. 1 letterh GDPR). However, this right has not yet been setout in more concrete terms. Companies, supervisory authorities and consumer organisations shouldwork together to develop standards for scoring services, which would help guarantee relevance andcomprehensibility. The Advisory Council furtherrecommends informing consumers of how theirpersonal score is to be interpreted against the distribution of score values among the population asa whole (e. g. does my score put me in the “upperthird”?).5. Prompt, free-of-charge notification should be provided – or at least offered as an option for consumers – in the event of major changes to a person’sscore (e. g. if the person slips into a lower category).Naturally, there are certain limitations to this: inorder to register a change in score, scoring serviceswould have to retain historical score values. Thereare many practical applications (such as fraud recognition or determining possible payment modalities) for which this option will not be available. Atbanks and insurance companies, scores are calculated on an ad-hoc basis. This means that no score

5 history is maintained, and potential changes are notapparent at the time the next “event” is registered.This proposal can therefore be implemented onlyat institutions where data collection is ongoing,e. g. credit scoring services and the Federal MotorTransport Authority in Flensburg (with its “Registerof Driver Fitness”, which already sends out such notifications).2. Fostering knowledgeand competenceAs recommended in the Advisory Council’s Digital Sovereignty report, NGOs, consumer protection organisationsand consumer protection projects should provide education on basic issues related to scoring in all its manifestations, as well as on the use of scoring in specificfields of business.1. For this purpose, the Federal Government shoulddevelop information and discussion materials aspart of its digitalisation strategy for the current parliamentary term, with the aim of improving skillson the part of consumers, multipliers and decision- makers. The underlying principles and quality aspects of scoring, as well as forms and causes of unequal treatment are just as much part of this basicknowledge as the rights enjoyed by those scored.2. Measures should be taken to foster the competencepeople require in order to take informed decisionsconcerning their participation in a scoring process.This includes having the skills to identify scoringservices and seek alternatives, as well as to verify,assess (e. g. is the information relevant to the consumer disclosed?) and utilise such services.3. Identifying and revealingdiscrimination1. The Advisory Council for Consumer Affairs recommends that consumer information rights, as set outin Article 15 para. 1 letter h of the GDPR, be strengthened. In particular, consumers should be able to ascertain how scores are distributed among differentgroups with different protected attributes (to theextent that this can be established by the servicesthemselves). This will allow consumers to provideevidence of algorithmic discrimination.2. The Advisory Council also recommends strengthening the position of supervisory authorities (seerecommendation 7).3. Furthermore, it recommends that associations begiven the right to pursue representative actions incases of discrimination through scoring.4. Ensuring that nontelematics based optionsremain available1. The Advisory Council for Consumer Affairs recommends the introduction of legal guarantees to maintain telematics-free options for those seeking insurance (especially motor vehicle liability insuranceand health insurance). In particular:2. Policyholders who do not use telematics-based tariffsmay not suffer substantial disadvantage compared tothe holders of telematics-based policies.3. Most members of the Advisory Council for Con sumer Affairs believe that telematics policies should

6 be self-financing and should not be offered at theexpense (even indirectly) of policyholders who douse telematics. Since solidarity objectives are relevant particularly in health insurance, steps wouldneed to be taken to prohibit cheaper telematicstariffs that exist only because they attract policyholders with above-average health and do not significantly reduce the expenses incurred by insurers.5. Ensuring score quality1. The Advisory Council for Consumer Affairs recommends that ambitious quality principles be developed on the basis of best practices. This shouldbe based on existing quality assurance initiativesfor algorithmic processes. These quality principlesshould be developed and updated (drafted, implemented, monitored) on a collaborative basis byindustry, supervisory authorities, consumer organisations and the market watchdogs of Germany’sconsumer advice centres.2. Scoring services operating in sensitive fields shouldbe obliged to file information with supervisory authorities that is verifiable in detail and reveals thehigh quality of their procedures. Only then will it bepossible to test scores for consumer fairness. Thisobligation would apply to scores which use statistical measures to predict behaviour (e. g. false positive rates, hit rate, gini coefficient, area under theROC) for the population as a whole and for relevantpopulation groups (by sex, age, education etc.). Thiswould also make it possible to identify discrimination and cases of questionable score quality.3. As the situation currently stands, scoring procedures that pursue objectives which have not beenappropriately identified to the public are prohibited by law. In addition to the role of supervisoryauthorities (see recommendation 7), consumer organisations or the market watchdogs of Germany’sconsumer advice centres could also apply theirexpertise and contribute to uncovering “falsely labelled” scores as well.4. The use of proxy variables, as for example in geoscoring, requires special justification (there mustbe a causal connection!) and must be subject to thescrutiny of the relevant supervisory authority. Theuse of proxy variables should be minimised. Whereproxy variables are used, plausible reasoning mustbe given as to their substantive connection with thetarget variable.6. Ensuring data quality1. When developing scores, a sufficient level of dataquality must be ensured and documented for supervisory authorities.2. Scoring services and users should enter into voluntary commitments to improve their data governance, in particular their data quality management,in accordance with the standards set in the qualityprinciples.3. In applying the procedure, measures must be takento ensure that data is accurate, complete and upto-date.4. In its report on Digital Sovereignty, the AdvisoryCouncil for Consumer Affairs already outlined theoption of a data dashboard, which would allowconsumers to scrutinise their own data. This wouldfacilitate consumer-oriented data management.The Advisory Council reaffirms its recommendation that this option be explored. Such explorationsshould cover current developments in the area ofsecure identity management via blockchain-basedsystems, which allow consumers to manage theirown identity data securely and definitively.5. The Advisory Council recommends that researchbe conducted promptly to appraise and, where applicable, improve the quality of data used in relevant scoring processes, with a particular focus onentity recognition. Where necessary, improvementsshould be made via statutory provisions. Measuresmust be taken to ensure that a score calculated for

7 a certain person is correctly assigned to that person.The duty for providers to inform individuals that theyare being scored (see recommendation for action 1)will serve to minimise the risk of identity mix-ups.In this regard there is clearly a conflict between theinterests of scoring services and users, on the onehand, and data protection interests on the other.For this reason the Advisory Council recommendsthat the Federal Government’s Data Ethics Commission discuss ways of improving entity recognitionand develop concrete recommendations.7. Improving oversight1. The Advisory Council for Consumer Affairs recommends that the Federal Government explore whethera digital agency (see the Advisory Council’s report on“Consumer Law 2.0”) could act as a competence centre to assist supervisory authorities in exercising theirmandates. This might consist, for example, in settingup a federal institute as a centre of method expertisefor quality assurance, which could also be used for“non-digital” purposes.2. The responsible supervisory authorities should beput in the position (both structurally and in partthrough salary improvements for specialists, especially in statistics and IT) to perform the aforementioned tasks. Developments at the Federal FinancialSupervisory Authority (BaFin) over the last few yearscould serve as good practice. The responsible supervisory authorities should be granted the considerable financial resources required for them to performthe aforementioned additional tasks and test concrete scoring services.3. To ensure that the present recommendations arepromptly implemented, the Advisory Council forConsumer Affairs proposes the creation of a taskforce at the level of the Federal Government (forexample at the Federal Chancellery) in order todevelop guidelines for the elaboration of qualityprinciples on the basis of existing procedures (e. g.at BaFin). This task force should be set up immediately after the Data Ethics Commission has finishedits work.8. Preventing “super scores”The Advisory Council for Consumer Affairs recommendsthat developments in China and in other countrieswhich are experimenting with “super scoring” are closely followed and analysed. In particular, public debate isrequired on the change in social values and structuresthat such systems entail.The development of “super scores” by internationalcommercial actors may also have an impact on Germany.Lawmakers and supervisory authorities should preparefor an examination of whether measures can and shouldbe taken to ensure that “super scores” cannot be offeredcommercially in Germany.The Advisory Council recommends that an examinationbe carried out into the extent to which existing instruments (especially purpose limitation and the “no tieins” rule) contained in the GDPR may also be used toprevent “super scores”.

8 Members and staff of the SVRVMembers of the SVRVProfessor Lucia Reisch (Chair)Professor of Intercultural Consumer Research andEuropean Consumer Policy at Copenhagen BusinessSchoolDr Daniela Büchel (Vice-Chair)Member of the Trade Germany Board, REWE Group,Managing Director of REWE Markt GmbH and of Penny-Markt GmbHProfessor Gerd GigerenzerDirector of the Harding Centre for Risk Literacy atthe Max Planck Institute for Human Development inBerlinHelga Zander-HayatMember of the Board of Management of NorthRhine-Westphalia Consumer Advice CentreProfessor Gesche JoostProfessor of Design Research at the University of FineArts, BerlinStaff of the SVRVHead of the Bureau:Thomas Fischer, M.A.Research staff of the Bureau:Johannes GerberdingDr Christian GrossDr Ariane KeitelSarah Sommer, M.A.Professor Hans-Wolfgang MicklitzProfessor of Economic Law at the European University Institute in FlorenceProfessor Andreas OehlerProfessor of Finance at the University of Bambergand Director of the University’s Research Centre forHousehold Finance and Financial LiteracyProfessor Kirsten Schlegel-MatthiesProfessor of Home Economics at the University ofPaderbornProfessor Gert G. WagnerMax Planck Fellow at the Max Planck Institute forHuman Development in Berlin, Research Associate atthe Alexander von Humboldt Institute for Internet andSociety, Berlin, and Senior Research Fellow for at theGerman Socio-Economic Panel Study at the GermanInstitute for Economic Research (DIW Berlin)

9TABLE OF CONTENTSTable of contentsAAbout this report I. Introduction II. Scores and scoring III. Objectives of the report Objective 1: Improve the information base and increase knowledge of scoring Objective 2: Broaden the empirical basis and address legal issues Objective 3: Suggest rules for consumer-friendly scoring BAreas for action:the state of research I. Transparency and comprehensibility 1. Transparency in predictive scoring 2. Transparency in behavioural scoring 3. Keeping transparency and comprehensibility of scoring systems on the agenda 4. Scoring transparency as a special form of algorithm transparency 5. Transparency as a condition for a social debate on scoring II. Non-discrimination and equal treatment 1. What is discrimination? 2. Discrimination through scoring input 3. Score quality and non discrimination 4. Undesirable unequal scoring-based treatment beyond discrimination 13141620202121252626272830323434353639III. Enforcement of rights 40IV. Score quality 411. Quality of the algorithm underlying a score 2. The utility of newer and more complex algorithms V. Baseline data 1. Accuracy, currency and completeness 2. Use of proxy variables 3. Weighting of input variables VI. Competing fairness criteria 41454646474850

10TABLE OF CONTENTSVII. Consumers and society: e xpectations, knowledge, competence and implications 521. Consumers’ expectations and acceptance of scoring 2. Knowledge and competence 3. Social implications 525457VIII. The danger of a super score 611. Scoring models abroad 2. Data accumulation and data trading 3. Repersonalisation of anonymised data 4. Aggregation of data into a super score 61656869CMarket survey: credit referenceagencies, motor i nsurancetelematics and health insurancepolicies 71I. Introduction and key issues II. Survey design 1. Overview of providers 2. The questionnaires III. Discussion of findings and highlighted consumer problems 1. Diffusion of scoring in the market segments under examination 2. Transparency 3. Score calculation and statistical quality 4. Behavioural effects 5. Discrimination 6. Aggregation of data and inclusion of new consumer attributes 7. Supervision 727374757676788084858788

11TABLE OF CONTENTSDPublic knowledge and acceptanceof scoring I. Preliminary study, 2017 II. Representative survey, 2018 1. Analysis of the findings 2. Multivariate regression analyses: presentation and discussion of findings 3. Population survey findings: general summary and conclusions 91929394106109EThe legal framework for scoring I. The basis in data privacy law 1. Profiling (Article 4(4) GDPR) 2. Automated individual decision-making (Article 22 GDPR) 3. Scoring of probability values (section 31 of the Federal Data Protection Act) II. Rules for specific areas of activity 1. The law governing standard business terms 2. The law governing insurance contracts and insurance supervision 3. Social insurance law and statutory health insurance III. Building blocks for a scoring regime 1. Regulating the ‘how’ of scoring versus regulating the ‘whether’ 2. Scoring regulation and algorithm regulation 3. Guaranteeing a defined score quality 4. Guaranteeing transparency and comprehensibility 5. Guaranteeing non-discrimination IV. Supervision 111113113115118124124125128129129130130132135138

12TABLE OF CONTENTS

About this reportAAbout thisreport13

14About this reportI.IntroductionUnder the heading of ‘scoring’, this report examines algorithmic decision-making processes involving direct consumer contact.1 In so doing, it follows on directly from thediscussions in previous reports from the Advisory Council for Consumer Affairs, particularly Consumer Rights 2.0(SVRV, 2016) and Digital Sovereignty (SVRV, 2017a). Thesubject of consumer scoring which was chosen for thisreport is assuming ever greater significance because of itstopicality and its increasingly wide use (see, for example,Christl and Spiekermann, 2016, and Mau, 2017). In manyspheres of people’s lives, increasingly complex methodsare being used to analyse consumers’ characteristicsand activity, predict their future behaviour or encouragethem to adopt modes of behaviour that will improve theirscore. The product of this analysis is an individual scorethat can serve as a basis for establishing: whether and on what conditions a consumercan obtain a mortgage, for example, how much discount a consumer can obtain fromhis or her motor insurance premium for gooddriving, and whether someone is taking sufficient preventiveaction to qualify for a bonus from his healthinsurer,and much more besides.These are examples from three major areas of life andconsumption, namely finance, mobility and health care,in which scoring is used today. These three areas havebeen selected for this report.In a market economy, scoring – particularly credit scoring – plays an important role in creating transparencyand trust between the two sides of the market, and, forexample, new score-based insurance products certainly offer benefits for consumers. Besides such beneficialeffects, however, scoring can also have unintended adverse effects.While the SVRV is fully aware of the potential of modernscoring systems, the focus of this report is on possiblerisks and ways of minimising them. Our specific goal isto examine what sort of form consumer-friendly scoring – which must first be defined – might take in termsof procedure and substance, what requirements it mustmeet in the light of consumer policy and how such consumer-friendly scoring can be politically and institutionally underpinned. These reflections are directly relevantto the regulation of algorithmic decision-making practices in general as well as to society’s assessment andregulation of artificial intelligence2 and to data ethics.3Scoring, the formalised rating of individuals with theaid of a numerical figure, has a certain tradition in ourculture; one need only think of school test and examination marks. Digitisation is now multiplying the means ofrating people and, therefore, increasing the risks arisingfrom such assessments. On the other hand, digitisationis also creating opportunities, because formalised scoring can be less discriminating as compared to informaldecisions taken by individuals, such as landlords oremployers. Numerous operations are embedded in anycomplex ‘decision-making architecture’ in which both1 The following are examples of other algorithmic decision-making processes that are not covered by this report:· consumer-related processes such as: 1. personalised vouchers in supermarkets and micro-targeting by online shops; these are based on information about theequipment with which users surf the Internet and about their browsing history to show personalised ads and offer personalised prices in order to induce users tomake purchases, to harness customers’ propensity to spend and to ensure customer retention by means of special offers (Hosell and Schleusener, 2016; ZanderHayat, Domurath and Gross, 2016; Zander-Hayat, Reisch and Steffen, 2016); 2. robo-advisers that assist in the selection of financial products (Oehler, Horn andWendt, 2016); 3. algorithm-controlled self-driving cars and other largely autonomously operating products, such as cleaning robots and robotic lawnmowers.· processes not directly relating to consumers such as: 1. people analytics (human-resources management; see, for example, Höller and Wede, 2018, written from atrade-union perspective), including applicant scoring (pre-employment screening and e-recruiting; see, for example, Christl, 2017); 2. predictive policing (see, forexample, Egbert, 2018, and Sommerer, 2017).2 On 28 June 2018, the Bundestag appointed a Study Commission on Artificial Intelligence, subtitled “Social Responsibility and Economic Potential”. The Bundestaghomepage states that “The task of the Commission is to formulate practical recommendations for dealing with artificial intelligence (AI). It is to be appointed withoutdelay and present its concluding report, including practical recommendations, after the 2020 sumer recess”. German text at z/560330, accessed on 17 August 2018.3 The Federal Government appointed a Data Ethics Commission to examine this issue. According to the homepage of the Federal Minbistry of the Interior, “There isgreat potential in the use of algorithms, artificial intelligence and digital innovations. At the same time, they raise numerous ethical and legal questions. ( ) Thepurpose of the Data Ethics Commission is to develop, on the basis of scientific and technical expertise, ethical guidelines for the protection of the individual, thepreservation of social cohesion and the maintenance of well-being in the information age. With the Federal Ministry of the Interior, Building and Community andthe Federal Ministry of Justice and Consumer Protection acting as the lead ministries, it will make practical recommendations by the summer of 2019 and proposeregulatory options.” German text at litik/ l, accessedon 12 August 2018.

15About this reporthuman decision-makers and machines are involved.Machines prioritise, sort and classify so as to focus theattention of human decision-makers. They stake out thearea within which human autonomy of action can unfold and prestructure human decision-making processes. Humans do not normally take decisions in a vacuum,so to speak; on the contrary, their decisions add anotherthread to an already complex social fabric.Before the potential of modern scoring systems can befully exploited, a number of conditions must be met toensure that the scoring is as consumer-fri

Berlin), Professor Stefan Lessmann (Humboldt University, Berlin), Professor Mario Martini (German University of Administrative Scienc-es in Speyer), Professor Steffen Mau (Humboldt University, Berlin), Professor Klaus-Robert Müller (Technical University of Berlin), Walter Palme