isbn-lcc-quick-convert.ipynb

{"metadata":{"kernelspec":{"display_name":"Python 3 (ipykernel)","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.7.12"}},"nbformat_minor":5,"nbformat":4,"cells":[{"cell_type":"markdown","source":"This Jupyter notebook is intended to allow you to upload your own csv file (containing one column and no header) into this temporary Binder space, create a new csv file with LCC in a new column, and save the output file. Use of the OCLC API must compy with the their [terms and conditions](https://www.oclc.org/research/areas/data-science/terms.html); this notebook is set up for demonstration only. \n\nThis Binder runs files from https://github.com/selenachau/binder/\nIt demonstrates a script that validates and cleans ISBN numbers and a script that pulls the most popular Library of Congress classification from ISBN numbers, using OCLC Classify. For more of a walkthrough and examples of extending this script further, see the Jypyter Notebook named isbn-lcc-walkthrough.ipynb\n\nOriginal/inspiration from https://github.com/mbelvadi/lcc_from_isbn, Copyright 2021 Melissa Belvadi Open Source License CC-BY-NC-SA granted. Changes: used OCLC Classify only, changed input/output method, and simplified to use a one column isbn list input.","metadata":{"tags":[]},"id":"4ae60090-7cda-4995-bc90-166e772bd3f6"},{"cell_type":"markdown","source":"Instructions\n\n1. Use the Binder upload icon (top left of browser window) to upload a csv file. It must be named isbns.csv, contain only one column of data, and have no header.\n\n2. Run the code block below using the Jupyter Notebook icons: click anywhere inside the block of code below, then click the play icon above.\n\n3. Wait some time and a new file named isbn-lcc.csv will appear in the file list. The time it takes for the code to run depends on the availability of the API service. Use the menu options to download your file.\n","metadata":{},"id":"38269e99-1989-4663-be99-81c70264b3cd"},{"cell_type":"code","source":"#!/usr/bin/env python3\n# -*- coding: utf-8 -*-\n\nimport logging\nimport isbnlib #https://pypi.org/project/isbnlib/\nimport requests #https://pypi.org/project/requests/\nfrom requests.utils import requote_uri\nimport xml.dom.pulldom\nimport xml.dom.minidom\nimport xml.sax.saxutils\nimport pandas as pd    \nfrom isbnlib import canonical \n\nUA = 'isbnlib (gzip)'\nmyheaders = {'User-Agent': UA}\nbase = 'http://classify.oclc.org/classify2/Classify?'\nsummaryBase = '&summary=true'\nsummaryInd = 'false'\nlogger = logging.getLogger(__name__)\nglobal parmvalue\nglobal parmtype\nparmtype=\"isbn\"\n\n# OCLC Classify2 method\ndef get_oclc_data(parmtype, parmvalue=\"\"):\n    global lcc_value\n    lcc_value = None\n    try:\n        nexturl = base + parmtype+\"=\" + requote_uri(parmvalue)+\"&summary=true\"\n        logger.debug(\"OCLC URL: {} \".format(nexturl))\n    except Exception as ue:\n        logger.error(\"OCLC URL encode failed: {}\".format(ue))\n        return None\n    else:\n        try:\n            r = requests.get(nexturl, headers=myheaders)\n            if not r.ok:\n                logger.error(\"OCLC Request returned http error: {}\".format(r.status_code))\n                return None\n        except Exception as e:\n            logger.error(\"OCLC URL request failed: {}\".format(e))\n            return None\n        else:\n            wq = r.text\n        xdoc = xml.dom.minidom.parseString(wq)\n    response = xdoc.getElementsByTagName('response')[0]\n    respCode = response.attributes[\"code\"].value\n    if respCode == '0' or respCode == '2':\n        recommendations = xdoc.getElementsByTagName('recommendations')[0]\n        if recommendations:\n            if len(xdoc.getElementsByTagName('lcc')) > 0:\n                local_lcc = recommendations.getElementsByTagName('lcc')[0]\n                if local_lcc:\n                    for mostPopular in local_lcc.getElementsByTagName('mostPopular'):\n                        nsfa = mostPopular.attributes[\"nsfa\"].value\n                        lcc_value = nsfa\n    elif respCode == '4':\n        works = xdoc.getElementsByTagName('works')[0]\n        logger.debug('Works found: ' + str(len(works.getElementsByTagName('work'))))\n        for work in works.getElementsByTagName('work'):\n            try:\n                m_wi = work.attributes[\"wi\"].value\n            except:\n                continue\n            else:\n                try:\n                    schemes = work.attributes[\"schemes\"].value\n                except:\n                    continue\n                if 'LCC' in schemes:\n                    logger.debug(f'going to try to get lcc using wi {m_wi}')\n                    lcc_value = get_oclc_data('wi',m_wi)\n                    break\n    elif respCode != '102':\n        logger.error(\"OCLC reporting odd error {}, check by hand: {}\".format(respCode,nexturl))\n    if lcc_value:\n        return lcc_value \n    else:\n        return None\n\n\ndef validate_json(data):\n    if str(data) == \"\":\n        logger.error(\"validate_json: returns False because no data in passed string: {}\".format(str(data)))\n        return False\n    return True\n\ndef fix_isbn(isbn):\n    lib_isbn = isbnlib.canonical(isbn)\n    if len(lib_isbn) in (10, 13):\n        if len(lib_isbn) == 10:\n            isgood = isbnlib.is_isbn10(lib_isbn)\n        else:\n            isgood = isbnlib.is_isbn13(lib_isbn)\n        if isgood:\n            return lib_isbn\n    if len(lib_isbn) < 10:\n        return None\n    lib_isbn = isbnlib.get_isbnlike(isbn)\n    if len(lib_isbn) < 10:\n        return None\n    lib_isbn = isbnlib.clean(lib_isbn)\n    if len(lib_isbn) < 10:\n        return None\n    lib_isbn = isbnlib.get_canonical_isbn(lib_isbn)\n    if len(lib_isbn) < 10:\n        return None\n    if not lib_isbn:\n        return None\n    if len(lib_isbn) in (10, 13):\n        if len(lib_isbn) == 10:\n            isgood = isbnlib.is_isbn10(lib_isbn)\n        else:\n            isgood = isbnlib.is_isbn13(lib_isbn)\n    else:\n        return None\n    if isgood:\n        return lib_isbn\n    else:\n        return None\n\n\n# create new function to be used only if ISBNs need validation.\ndef clean_isbn(row):\n    x = canonical(row[0])\n    return get_oclc_data(parmtype, x)\n\n\ndf = pd.read_csv('isbns.csv', dtype=\"string\", names=['isbn'])\ndf['lcc'] = df.apply(clean_isbn, axis=1)\n\ndf.to_csv('isbn-lcc.csv', encoding='utf-8', index=False)\n\n","metadata":{},"execution_count":1,"outputs":[],"id":"e53acdce-42c3-480d-bae2-96f94ca8f529"},{"cell_type":"markdown","source":"Copyright 2022 Selena Chau CC-BY-NC-SA granted","metadata":{},"id":"b6eabaab-c2f6-49f5-96e3-160d6619ef95"}]}