Name server list contains incorrect encodings

[peterdesmet]

In the interim and processed data of the name server species, some names are written with incorrect encoding characters, such as Agaricus xantholepis F.H. Møller, 1952 instead of Agaricus xantholepis F.H. Møller, 1952 (e.g. in this file).

The problem is related to the way the GBIF API is requested. With the requests library (used in e.g. the gbif_name_matching utility), there are no issues:

import requests
results = requests.get('http://api.gbif.org/v1/species/match?verbose=False&strict=true&name=Agaricus+xantholepis')
results.json()

Returns:

{'canonicalName': 'Agaricus xantholepis',
 'class': 'Agaricomycetes',
 'classKey': 186,
 'confidence': 99,
 'family': 'Agaricaceae',
 'familyKey': 4170,
 'genus': 'Agaricus',
 'genusKey': 7893259,
 'kingdom': 'Fungi',
 'kingdomKey': 5,
 'matchType': 'EXACT',
 'order': 'Agaricales',
 'orderKey': 1499,
 'phylum': 'Basidiomycota',
 'phylumKey': 34,
 'rank': 'SPECIES',
 'scientificName': 'Agaricus xantholepis F.H. Møller, 1952',
 'species': 'Agaricus xantholepis',
 'speciesKey': 5243402,
 'status': 'ACCEPTED',
 'synonym': False,
 'usageKey': 5243402}

But the name server list is so big, that an asynchronous request is necessary. This is implemented in this notebook. Apparently the asyncio or other library that is used to request GBIF does not handle the result as UTF8?