Upates for new release

Updated teststatus

.github/ISSUE_TEMPLATE/bug_report.md

@@ -31,7 +31,7 @@ If applicable, add screenshots to help explain your problem.
  - OS: [e.g. Windows 10/Raspberry Pi OS]
  - Python version: [e.g. python2.7]
  - Calibre-Web version: [e.g. 0.6.8 or 087c4c59 (git rev-parse --short HEAD)]:
- - Docker container: [None/Technosoft2000/LinuxServer]:
+ - Docker container: [None/LinuxServer]:
  - Special Hardware: [e.g. Rasperry Pi Zero]
  - Browser: [e.g. Chrome 83.0.4103.97, Safari 13.3.7, Firefox 68.0.1 ESR]
 

README.md

@@ -19,7 +19,7 @@ Calibre-Web is a web app providing a clean interface for browsing, reading and d
 - full graphical setup
 - User management with fine-grained per-user permissions
 - Admin interface
-- User Interface in brazilian, czech, dutch, english, finnish, french, german, greek, hungarian, italian, japanese, khmer, polish, russian, simplified chinese, spanish, swedish, turkish, ukrainian
+- User Interface in brazilian, czech, dutch, english, finnish, french, german, greek, hungarian, italian, japanese, khmer, korean, polish, russian, simplified and traditional chinese, spanish, swedish, turkish, ukrainian
 - OPDS feed for eBook reader apps 
 - Filter and search by titles, authors, tags, series and language
 - Create a custom book collection (shelves)
@@ -37,9 +37,9 @@ Calibre-Web is a web app providing a clean interface for browsing, reading and d
 - "Magic Link" login to make it easy to log on eReaders
 - Login via LDAP, google/github oauth and via proxy authentication
 
-## Quick start
+## Installation
 
-#### Install via pip
+#### Installation via pip (recommended)
 1. Install calibre web via pip with the command `pip install calibreweb` (Depending on your OS and or distro the command could also be `pip3`). 
 2. Optional features can also be installed via pip, please refer to [this page](https://github.com/janeczku/calibre-web/wiki/Dependencies-in-Calibre-Web-Linux-Windows) for details 
 3. Calibre-Web can be started afterwards by typing `cps` or `python3 -m cps` 
@@ -48,21 +48,24 @@ Calibre-Web is a web app providing a clean interface for browsing, reading and d
 1. Install dependencies by running `pip3 install --target vendor -r requirements.txt` (python3.x). Alternativly set up a python virtual environment.
 2. Execute the command: `python3 cps.py` (or `nohup python3 cps.py` - recommended if you want to exit the terminal window)
 
+Issues with Ubuntu:
+Please note that running the above install command can fail on some versions of Ubuntu, saying `"can't combine user with prefix"`. This is a [known bug](https://github.com/pypa/pip/issues/3826) and can be remedied by using the command `pip install --system --target vendor -r requirements.txt` instead.
+
+## Quick start
+
 Point your browser to `http://localhost:8083` or `http://localhost:8083/opds` for the OPDS catalog
 Set `Location of Calibre database` to the path of the folder where your Calibre library (metadata.db) lives, push "submit" button\
 Optionally a Google Drive can be used to host the calibre library [-> Using Google Drive integration](https://github.com/janeczku/calibre-web/wiki/Configuration#using-google-drive-integration)
 Go to Login page
 
-**Default admin login:**\
+#### Default admin login:
 *Username:* admin\
 *Password:* admin123
 
-**Issues with Ubuntu:**
-Please note that running the above install command can fail on some versions of Ubuntu, saying `"can't combine user with prefix"`. This is a [known bug](https://github.com/pypa/pip/issues/3826) and can be remedied by using the command `pip install --system --target vendor -r requirements.txt` instead.
 
 ## Requirements
 
-python 3.x+
+python 3.5+
 
 Optionally, to enable on-the-fly conversion from one ebook format to another when using the send-to-kindle feature, or during editing of ebooks metadata:
 
@@ -72,14 +75,7 @@ Optionally, to enable on-the-fly conversion from one ebook format to another whe
 
 ## Docker Images
 
-Pre-built Docker images are available in these Docker Hub repositories:
+A pre-built Docker image is available in these Docker Hub repository (maintained by the LinuxServer team):
-
-#### **Technosoft2000 - x64**
-+ Docker Hub - [https://hub.docker.com/r/technosoft2000/calibre-web](https://hub.docker.com/r/technosoft2000/calibre-web)
-+ Github - [https://github.com/Technosoft2000/docker-calibre-web](https://github.com/Technosoft2000/docker-calibre-web) 
-
-    Includes the Calibre `ebook-convert` binary.
-    + The "path to convertertool" should be set to `/opt/calibre/ebook-convert`
 
 #### **LinuxServer - x64, armhf, aarch64**
 + Docker Hub - [https://hub.docker.com/r/linuxserver/calibre-web](https://hub.docker.com/r/linuxserver/calibre-web)

SECURITY.md

@@ -3,3 +3,37 @@
 ## Reporting a Vulnerability
 
 Please report security issues to ozzie.fernandez.isaacs@googlemail.com
+
+## Supported Versions
+
+To receive fixes for security vulnerabilities it is required to always upgrade to the latest version of Calibre-Web. See https://github.com/janeczku/calibre-web/releases/latest for the latest release.
+
+## History
+
+| Fixed in      | Description                                                                                                        |CVE number |
+|---------------|--------------------------------------------------------------------------------------------------------------------|---------|
+| 3rd July 2018 | Guest access acts as a backdoor                                                                                    ||
+| V 0.6.7       | Hardcoded secret key for sessions                                                                                  |CVE-2020-12627 |
+| V 0.6.13      | Calibre-Web Metadata cross site scripting                                                                          |CVE-2021-25964|
+| V 0.6.13      | Name of Shelves are only visible to users who can access the corresponding shelf Thanks to @ibarrionuevo           ||
+| V 0.6.13      | JavaScript could get executed in the description field. Thanks to @ranjit-git  and Hagai Wechsler (WhiteSource)    ||
+| V 0.6.13      | JavaScript could get executed in a custom column of type "comment" field                                           ||
+| V 0.6.13      | JavaScript could get executed after converting a book to another format with a title containing javascript code    ||
+| V 0.6.13      | JavaScript could get executed after converting a book to another format with a username containing javascript code ||
+| V 0.6.13      | JavaScript could get executed in the description series, categories or publishers title                            ||
+| V 0.6.13      | JavaScript could get executed  in the shelf title                                                                  ||
+| V 0.6.13      | Login with the old session cookie after logout. Thanks to @ibarrionuevo                                            ||
+| V 0.6.14      | CSRF was possible. Thanks to @mik317 and Hagai Wechsler (WhiteSource)                                              |CVE-2021-25965|
+| V 0.6.14      | Migrated some routes to POST-requests (CSRF protection). Thanks to @scara31                                        ||
+| V 0.6.15      | Fix for "javascript:" script links in identifier. Thanks to @scara31                                               ||
+| V 0.6.15      | Cross-Site Scripting vulnerability on uploaded cover file names. Thanks to @ibarrionuevo                           ||
+| V 0.6.15      | Creating public shelfs is now denied if user is missing the edit public shelf right. Thanks to @ibarrionuevo       ||
+| V 0.6.15      | Changed error message in case of trying to delete a shelf unauthorized. Thanks to @ibarrionuevo                    ||
+| V 0.6.16      | JavaScript could get executed on authors page. Thanks to @alicaz                                                   ||
+| V 0.6.16      | Localhost can no longer be used to upload covers. Thanks to @scara31                                               ||
+| V 0.6.16      | Another case where public shelfs could be created without permission is prevented. Thanks to @ibarrionuevo         ||
+
+
+## Staement regarding Log4j (CVE-2021-44228 and related)
+
+Calibre-web is not affected by bugs related to Log4j. Calibre-Web is a python program, therefore not using Java, and not using the Java logging feature log4j. 

cps.py

@@ -17,18 +17,13 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import absolute_import, division, print_function, unicode_literals
 import sys
 import os
 
 
 # Insert local directories into path
-if sys.version_info < (3, 0):
+sys.path.append(os.path.dirname(os.path.abspath(__file__)))
-    sys.path.append(os.path.dirname(os.path.abspath(__file__.decode('utf-8'))))
+sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'vendor'))
-    sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__.decode('utf-8'))), 'vendor'))
-else:
-    sys.path.append(os.path.dirname(os.path.abspath(__file__)))
-    sys.path.append(os.path.join(os.path.dirname(os.path.abspath(__file__)), 'vendor'))
 
 
 from cps import create_app
@@ -42,13 +37,14 @@ from cps.admin import admi
 from cps.gdrive import gdrive
 from cps.editbooks import editbook
 from cps.remotelogin import remotelogin
+from cps.search_metadata import meta
 from cps.error_handler import init_errorhandler
 
 try:
     from cps.kobo import kobo, get_kobo_activated
     from cps.kobo_auth import kobo_auth
     kobo_available = get_kobo_activated()
-except ImportError:
+except (ImportError, AttributeError):   # Catch also error for not installed flask-WTF (missing csrf decorator)
     kobo_available = False
 
 try:
@@ -70,6 +66,7 @@ def main():
     app.register_blueprint(shelf)
     app.register_blueprint(admi)
     app.register_blueprint(remotelogin)
+    app.register_blueprint(meta)
     app.register_blueprint(gdrive)
     app.register_blueprint(editbook)
     if kobo_available:

cps/MyLoginManager.py

@@ -0,0 +1,34 @@
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2018-2019 OzzieIsaacs, cervinko, jkrehm, bodybybuddha, ok11,
+#                            andy29485, idalin, Kyosfonica, wuqi, Kennyl, lemmsh,
+#                            falgh1, grunjol, csitko, ytils, xybydy, trasba, vrabe,
+#                            ruben-herold, marblepebble, JackED42, SiphonSquirrel,
+#                            apetresc, nanu-c, mutschler, GammaC0de, vuolter
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+from flask_login import LoginManager
+from flask import session
+
+
+class MyLoginManager(LoginManager):
+    def _session_protection_failed(self):
+        sess = session._get_current_object()
+        ident = self._session_identifier_generator()
+        if(sess and not (len(sess) == 1 and sess.get('csrf_token', None))) and ident != sess.get('_id', None):
+            return super(). _session_protection_failed()
+        return False

cps/__init__.py

@@ -19,8 +19,8 @@
 #
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
+__package__ = "cps"
 
-from __future__ import division, print_function, unicode_literals
 import sys
 import os
 import mimetypes
@@ -29,13 +29,14 @@ from babel import Locale as LC
 from babel import negotiate_locale
 from babel.core import UnknownLocaleError
 from flask import Flask, request, g
-from flask_login import LoginManager
+from .MyLoginManager import MyLoginManager
 from flask_babel import Babel
 from flask_principal import Principal
 
 from . import config_sql, logger, cache_buster, cli, ub, db
 from .reverseproxy import ReverseProxied
 from .server import WebServer
+from .dep_check import dependency_check
 
 try:
     import lxml
@@ -43,6 +44,12 @@ try:
 except ImportError:
     lxml_present = False
 
+try:
+    from flask_wtf.csrf import CSRFProtect
+    wtf_present = True
+except ImportError:
+    wtf_present = False
+
 mimetypes.init()
 mimetypes.add_type('application/xhtml+xml', '.xhtml')
 mimetypes.add_type('application/epub+zip', '.epub')
@@ -61,20 +68,29 @@ mimetypes.add_type('application/mp4', '.m4a')
 mimetypes.add_type('application/mp4', '.m4b')
 mimetypes.add_type('application/ogg', '.ogg')
 mimetypes.add_type('application/ogg', '.oga')
+mimetypes.add_type('text/css', '.css')
+mimetypes.add_type('text/javascript; charset=UTF-8', '.js')
 
 app = Flask(__name__)
 app.config.update(
     SESSION_COOKIE_HTTPONLY=True,
     SESSION_COOKIE_SAMESITE='Lax',
     REMEMBER_COOKIE_SAMESITE='Lax',  # will be available in flask-login 0.5.1 earliest
+    WTF_CSRF_SSL_STRICT=False
 )
 
 
-lm = LoginManager()
+lm = MyLoginManager()
 lm.login_view = 'web.login'
 lm.anonymous_user = ub.Anonymous
 lm.session_protection = 'strong'
 
+if wtf_present:
+    csrf = CSRFProtect()
+    csrf.init_app(app)
+else:
+    csrf = None
+
 ub.init_db(cli.settingspath)
 # pylint: disable=no-member
 config = config_sql.load_configuration(ub.session)
@@ -86,6 +102,7 @@ _BABEL_TRANSLATIONS = set()
 
 log = logger.create()
 
+
 from . import services
 
 db.CalibreDB.update_config(config)
@@ -100,17 +117,24 @@ def create_app():
             '*** Python2 is EOL since end of 2019, this version of Calibre-Web is no longer supporting Python2, please update your installation to Python3 ***')
         print(
             '*** Python2 is EOL since end of 2019, this version of Calibre-Web is no longer supporting Python2, please update your installation to Python3 ***')
+        web_server.stop(True)
         sys.exit(5)
     if not lxml_present:
         log.info('*** "lxml" is needed for calibre-web to run. Please install it using pip: "pip install lxml" ***')
         print('*** "lxml" is needed for calibre-web to run. Please install it using pip: "pip install lxml" ***')
+        web_server.stop(True)
         sys.exit(6)
+    if not wtf_present:
+        log.info('*** "flask-WTF" is needed for calibre-web to run. Please install it using pip: "pip install flask-WTF" ***')
+        print('*** "flask-WTF" is needed for calibre-web to run. Please install it using pip: "pip install flask-WTF" ***')
+        web_server.stop(True)
+        sys.exit(7)
+    for res in dependency_check() + dependency_check(True):
+        log.info('*** "{}" version does not fit the requirements. Should: {}, Found: {}, please consider installing required version ***'
+            .format(res['name'],
+                 res['target'],
+                 res['found']))
     app.wsgi_app = ReverseProxied(app.wsgi_app)
-    # For python2 convert path to unicode
-    if sys.version_info < (3, 0):
-        app.static_folder = app.static_folder.decode('utf-8')
-        app.root_path = app.root_path.decode('utf-8')
-        app.instance_path = app.instance_path.decode('utf-8')
 
     if os.environ.get('FLASK_DEBUG'):
         cache_buster.init_cache_busting(app)
@@ -139,7 +163,6 @@ def create_app():
 def get_locale():
     # if a user is logged in, use the locale from the user settings
     user = getattr(g, 'user', None)
-    # user = None
     if user is not None and hasattr(user, "locale"):
         if user.name != 'Guest':   # if the account is the guest account bypass the config lang settings
             return user.locale
@@ -160,6 +183,7 @@ def get_timezone():
     user = getattr(g, 'user', None)
     return user.timezone if user else None
 
+
 from .updater import Updater
 updater_thread = Updater()
 updater_thread.start()

cps/about.py

@@ -20,7 +20,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import sys
 import platform
 import sqlite3
@@ -29,9 +28,14 @@ from collections import OrderedDict
 import babel, pytz, requests, sqlalchemy
 import werkzeug, flask, flask_login, flask_principal, jinja2
 from flask_babel import gettext as _
+try:
+    from flask_wtf import __version__ as flaskwtf_version
+except ImportError:
+    flaskwtf_version = _(u'not installed')
 
-from . import db, calibre_db, converter, uploader, server, isoLanguages, constants
+from . import db, calibre_db, converter, uploader, server, isoLanguages, constants, gdriveutils, dep_check
 from .render_template import render_title_template
+
 try:
     from flask_login import __version__ as flask_loginVersion
 except ImportError:
@@ -64,17 +68,33 @@ from . import services
 
 about = flask.Blueprint('about', __name__)
 
+ret = dict()
+req = dep_check.load_dependencys(False)
+opt = dep_check.load_dependencys(True)
+for i in (req + opt):
+    ret[i[1]] = i[0]
 
-_VERSIONS = OrderedDict(
+if constants.NIGHTLY_VERSION[0] == "$Format:%H$":
+    calibre_web_version = constants.STABLE_VERSION['version']
+else:
+    calibre_web_version = (constants.STABLE_VERSION['version'] + ' - '
+                    + constants.NIGHTLY_VERSION[0].replace('%','%%') + ' - '
+                    + constants.NIGHTLY_VERSION[1].replace('%','%%'))
+if getattr(sys, 'frozen', False):
+    calibre_web_version += " - Exe-Version"
+elif constants.HOME_CONFIG:
+    calibre_web_version += " - pyPi"
+
+if not ret:
+    _VERSIONS = OrderedDict(
         Platform = '{0[0]} {0[2]} {0[3]} {0[4]} {0[5]}'.format(platform.uname()),
         Python=sys.version,
-    Calibre_Web=constants.STABLE_VERSION['version'] + ' - '
+        Calibre_Web=calibre_web_version,
-                + constants.NIGHTLY_VERSION[0].replace('%','%%') + ' - '
-                + constants.NIGHTLY_VERSION[1].replace('%','%%'),
         WebServer=server.VERSION,
         Flask=flask.__version__,
         Flask_Login=flask_loginVersion,
         Flask_Principal=flask_principal.__version__,
+        Flask_WTF=flaskwtf_version,
         Werkzeug=werkzeug.__version__,
         Babel=babel.__version__,
         Jinja2=jinja2.__version__,
@@ -84,17 +104,29 @@ _VERSIONS = OrderedDict(
         SQLite=sqlite3.sqlite_version,
         iso639=isoLanguages.__version__,
         pytz=pytz.__version__,
-    Unidecode = unidecode_version,
+        Unidecode=unidecode_version,
-    Scholarly = scholarly_version,
+        Scholarly=scholarly_version,
-    Flask_SimpleLDAP =  u'installed' if bool(services.ldap) else None,
+        Flask_SimpleLDAP=u'installed' if bool(services.ldap) else None,
-    python_LDAP = services.ldapVersion if bool(services.ldapVersion) else None,
+        python_LDAP=services.ldapVersion if bool(services.ldapVersion) else None,
-    Goodreads = u'installed' if bool(services.goodreads_support) else None,
+        Goodreads=u'installed' if bool(services.goodreads_support) else None,
-    jsonschema = services.SyncToken.__version__  if bool(services.SyncToken) else None,
+        jsonschema=services.SyncToken.__version__ if bool(services.SyncToken) else None,
-    flask_dance = flask_danceVersion,
+        flask_dance=flask_danceVersion,
-    greenlet = greenlet_Version
+        greenlet=greenlet_Version
-)
+    )
-_VERSIONS.update(uploader.get_versions())
+    _VERSIONS.update(gdriveutils.get_versions())
-
+    _VERSIONS.update(uploader.get_versions(True))
+else:
+    _VERSIONS = OrderedDict(
+        Platform = '{0[0]} {0[2]} {0[3]} {0[4]} {0[5]}'.format(platform.uname()),
+        Python = sys.version,
+        Calibre_Web=calibre_web_version,
+        Werkzeug = werkzeug.__version__,
+        Jinja2=jinja2.__version__,
+        pySqlite = sqlite3.version,
+        SQLite = sqlite3.sqlite_version,
+    )
+    _VERSIONS.update(ret)
+    _VERSIONS.update(uploader.get_versions(False))
 
 def collect_stats():
     _VERSIONS['ebook converter'] = _(converter.get_calibre_version())
@@ -111,5 +143,3 @@ def stats():
     series = calibre_db.session.query(db.Series).count()
     return render_title_template('stats.html', bookcounter=counter, authorcounter=authors, versions=collect_stats(),
                                  categorycounter=categorys, seriecounter=series, title=_(u"Statistics"), page="stat")
-
-

cps/admin.py

@@ -20,7 +20,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import os
 import re
 import base64
@@ -41,7 +40,7 @@ from sqlalchemy.exc import IntegrityError, OperationalError, InvalidRequestError
 from sqlalchemy.sql.expression import func, or_, text
 
 from . import constants, logger, helper, services
-from . import db, calibre_db, ub, web_server, get_locale, config, updater_thread, babel, gdriveutils
+from . import db, calibre_db, ub, web_server, get_locale, config, updater_thread, babel, gdriveutils, kobo_sync_status
 from .helper import check_valid_domain, send_test_mail, reset_password, generate_password_hash, check_email, \
     valid_email, check_username
 from .gdriveutils import is_gdrive_ready, gdrive_support
@@ -130,11 +129,11 @@ def admin_forbidden():
     abort(403)
 
 
-@admi.route("/shutdown")
+@admi.route("/shutdown", methods=["POST"])
 @login_required
 @admin_required
 def shutdown():
-    task = int(request.args.get("parameter").strip())
+    task = request.get_json().get('parameter', -1)
     showtext = {}
     if task in (0, 1):  # valid commandos received
         # close all database connections
@@ -146,7 +145,7 @@ def shutdown():
         else:
             showtext['text'] = _(u'Performing shutdown of server, please close window')
         # stop gevent/tornado server
-        web_server.stop(task==0)
+        web_server.stop(task == 0)
         return json.dumps(showtext)
 
     if task == 2:
@@ -236,8 +235,12 @@ def view_configuration():
         .filter(and_(db.Custom_Columns.datatype == 'bool', db.Custom_Columns.mark_for_delete == 0)).all()
     restrict_columns = calibre_db.session.query(db.Custom_Columns)\
         .filter(and_(db.Custom_Columns.datatype == 'text', db.Custom_Columns.mark_for_delete == 0)).all()
+    languages = calibre_db.speaking_language()
+    translations = [LC('en')] + babel.list_translations()
     return render_title_template("config_view_edit.html", conf=config, readColumns=read_column,
                                  restrictColumns=restrict_columns,
+                                 languages=languages,
+                                 translations=translations,
                                  title=_(u"UI Configuration"), page="uiconfig")
 
 @admi.route("/admin/usertable")
@@ -515,16 +518,12 @@ def check_valid_restricted_column(column):
     return True
 
 
-
 @admi.route("/admin/viewconfig", methods=["POST"])
 @login_required
 @admin_required
 def update_view_configuration():
     to_save = request.form.to_dict()
 
-    # _config_string = lambda x: config.set_from_dictionary(to_save, x, lambda y: y.strip() if y else y)
-    # _config_int = lambda x: config.set_from_dictionary(to_save, x, int)
-
     _config_string(to_save, "config_calibre_web_title")
     _config_string(to_save, "config_columns_to_ignore")
     if _config_string(to_save, "config_title_regex"):
@@ -546,6 +545,8 @@ def update_view_configuration():
     _config_int(to_save, "config_random_books")
     _config_int(to_save, "config_books_per_page")
     _config_int(to_save, "config_authors_max")
+    _config_string(to_save, "config_default_language")
+    _config_string(to_save, "config_default_locale")
 
 
     config.config_default_role = constants.selected_roles(to_save)
@@ -589,6 +590,8 @@ def load_dialogtexts(element_id):
         texts["main"] = _('Are you sure you want to change shelf sync behavior for the selected user(s)?')
     elif element_id == "db_submit":
         texts["main"] = _('Are you sure you want to change Calibre library location?')
+    elif element_id == "btnfullsync":
+        texts["main"] = _("Are you sure you want delete Calibre-Web's sync database to force a full sync with your Kobo Reader?")
     return json.dumps(texts)
 
 
@@ -753,7 +756,12 @@ def prepare_tags(user, action, tags_name, id_list):
     return ",".join(saved_tags_list)
 
 
-@admi.route("/ajax/addrestriction/<int:res_type>", defaults={"user_id": 0}, methods=['POST'])
+@admi.route("/ajax/addrestriction/<int:res_type>", methods=['POST'])
+@login_required
+@admin_required
+def add_user_0_restriction(res_type):
+    return add_restriction(res_type, 0)
+
 @admi.route("/ajax/addrestriction/<int:res_type>/<int:user_id>", methods=['POST'])
 @login_required
 @admin_required
@@ -800,7 +808,13 @@ def add_restriction(res_type, user_id):
     return ""
 
 
-@admi.route("/ajax/deleterestriction/<int:res_type>", defaults={"user_id": 0}, methods=['POST'])
+@admi.route("/ajax/deleterestriction/<int:res_type>", methods=['POST'])
+@login_required
+@admin_required
+def delete_user_0_restriction(res_type):
+    return delete_restriction(res_type, 0)
+
+
 @admi.route("/ajax/deleterestriction/<int:res_type>/<int:user_id>", methods=['POST'])
 @login_required
 @admin_required
@@ -888,10 +902,18 @@ def list_restriction(res_type, user_id):
     else:
         json_dumps = ""
     js = json.dumps(json_dumps)
-    response = make_response(js) #.replace("'", '"')
+    response = make_response(js)
     response.headers["Content-Type"] = "application/json; charset=utf-8"
     return response
 
+@admi.route("/ajax/fullsync", methods=["POST"])
+@login_required
+def ajax_fullsync():
+    count = ub.session.query(ub.KoboSyncedBooks).filter(current_user.id == ub.KoboSyncedBooks.user_id).delete()
+    message = _("{} sync entries deleted").format(count)
+    ub.session_commit(message)
+    return Response(json.dumps([{"type": "success", "message": message}]), mimetype='application/json')
+
 
 @admi.route("/ajax/pathchooser/")
 @login_required
@@ -1185,11 +1207,20 @@ def _db_configuration_update_helper():
         if not calibre_db.setup_db(to_save['config_calibre_dir'], ub.app_DB_path):
             return _db_configuration_result(_('DB Location is not Valid, Please Enter Correct Path'),
                                             gdrive_error)
+        # if db changed -> delete shelfs, delete download books, delete read books, kobo sync...
+        ub.session.query(ub.Downloads).delete()
+        ub.session.query(ub.ArchivedBook).delete()
+        ub.session.query(ub.ReadBook).delete()
+        ub.session.query(ub.BookShelf).delete()
+        ub.session.query(ub.Bookmark).delete()
+        ub.session.query(ub.KoboReadingState).delete()
+        ub.session.query(ub.KoboStatistics).delete()
+        ub.session.query(ub.KoboSyncedBooks).delete()
+        ub.session_commit()
         _config_string(to_save, "config_calibre_dir")
         calibre_db.update_config(config)
         if not os.access(os.path.join(config.config_calibre_dir, "metadata.db"), os.W_OK):
             flash(_(u"DB is not Writeable"), category="warning")
-            # warning = {'type': "warning", 'message': _(u"DB is not Writeable")}
     config.save()
     return _db_configuration_result(None, gdrive_error)
 
@@ -1198,7 +1229,7 @@ def _configuration_update_helper():
     to_save = request.form.to_dict()
     try:
         reboot_required |= _config_int(to_save, "config_port")
-
+        reboot_required |= _config_string(to_save, "config_trustedhosts")
         reboot_required |= _config_string(to_save, "config_keyfile")
         if config.config_keyfile and not os.path.isfile(config.config_keyfile):
             return _configuration_result(_('Keyfile Location is not Valid, Please Enter Correct Path'))
@@ -1208,6 +1239,7 @@ def _configuration_update_helper():
             return _configuration_result(_('Certfile Location is not Valid, Please Enter Correct Path'))
 
         _config_checkbox_int(to_save, "config_uploading")
+        _config_checkbox_int(to_save, "config_unicode_filename")
         # Reboot on config_anonbrowse with enabled ldap, as decoraters are changed in this case
         reboot_required |= (_config_checkbox_int(to_save, "config_anonbrowse")
                              and config.config_login_type == constants.LOGIN_LDAP)
@@ -1310,6 +1342,8 @@ def _db_configuration_result(error_flash=None, gdrive_error=None):
         log.error(error_flash)
         config.load()
         flash(error_flash, category="error")
+    elif request.method == "POST" and not gdrive_error:
+        flash(_("Database Settings updated"), category="success")
 
     return render_title_template("config_db.html",
                                  config=config,
@@ -1344,7 +1378,9 @@ def _handle_new_user(to_save, content, languages, translations, kobo_support):
             raise Exception(_(u"E-mail is not from valid domain"))
     except Exception as ex:
         flash(str(ex), category="error")
-        return render_title_template("user_edit.html", new_user=1, content=content, translations=translations,
+        return render_title_template("user_edit.html", new_user=1, content=content,
+                                     config=config,
+                                     translations=translations,
                                      languages=languages, title=_(u"Add new user"), page="newuser",
                                      kobo_support=kobo_support, registered_oauth=oauth_check)
     try:
@@ -1379,16 +1415,25 @@ def _delete_user(content):
             for us in ub.session.query(ub.Shelf).filter(content.id == ub.Shelf.user_id):
                 ub.session.query(ub.BookShelf).filter(us.id == ub.BookShelf.shelf).delete()
             ub.session.query(ub.Shelf).filter(content.id == ub.Shelf.user_id).delete()
+            ub.session.query(ub.Bookmark).filter(content.id == ub.Bookmark.user_id).delete()
             ub.session.query(ub.User).filter(ub.User.id == content.id).delete()
+            ub.session.query(ub.ArchivedBook).filter(ub.ArchivedBook.user_id == content.id).delete()
+            ub.session.query(ub.RemoteAuthToken).filter(ub.RemoteAuthToken.user_id == content.id).delete()
+            ub.session.query(ub.User_Sessions).filter(ub.User_Sessions.user_id == content.id).delete()
+            ub.session.query(ub.KoboSyncedBooks).filter(ub.KoboSyncedBooks.user_id == content.id).delete()
+            # delete KoboReadingState and all it's children
+            kobo_entries = ub.session.query(ub.KoboReadingState).filter(ub.KoboReadingState.user_id == content.id).all()
+            for kobo_entry in kobo_entries:
+                ub.session.delete(kobo_entry)
             ub.session_commit()
-            log.info(u"User {} deleted".format(content.name))
+            log.info("User {} deleted".format(content.name))
-            return(_(u"User '%(nick)s' deleted", nick=content.name))
+            return(_("User '%(nick)s' deleted", nick=content.name))
         else:
-            log.warning(_(u"Can't delete Guest User"))
+            log.warning(_("Can't delete Guest User"))
-            raise Exception(_(u"Can't delete Guest User"))
+            raise Exception(_("Can't delete Guest User"))
     else:
-        log.warning(u"No admin user remaining, can't delete user")
+        log.warning("No admin user remaining, can't delete user")
-        raise Exception(_(u"No admin user remaining, can't delete user"))
+        raise Exception(_("No admin user remaining, can't delete user"))
 
 
 def _handle_edit_user(to_save, content, languages, translations, kobo_support):
@@ -1428,8 +1473,13 @@ def _handle_edit_user(to_save, content, languages, translations, kobo_support):
         else:
             content.sidebar_view &= ~constants.DETAIL_RANDOM
 
+        old_state = content.kobo_only_shelves_sync
         content.kobo_only_shelves_sync = int(to_save.get("kobo_only_shelves_sync") == "on") or 0
-
+        # 1 -> 0: nothing has to be done
+        # 0 -> 1: all synced books have to be added to archived books, + currently synced shelfs
+        # which don't have to be synced have to be removed (added to Shelf archive)
+        if old_state == 0 and content.kobo_only_shelves_sync == 1:
+            kobo_sync_status.update_on_sync_shelfs(content.id)
         if to_save.get("default_language"):
             content.default_language = to_save["default_language"]
         if to_save.get("locale"):
@@ -1454,6 +1504,7 @@ def _handle_edit_user(to_save, content, languages, translations, kobo_support):
                                          kobo_support=kobo_support,
                                          new_user=0,
                                          content=content,
+                                         config=config,
                                          registered_oauth=oauth_check,
                                          title=_(u"Edit User %(nick)s", nick=content.name),
                                          page="edituser")
@@ -1485,7 +1536,10 @@ def new_user():
     else:
         content.role = config.config_default_role
         content.sidebar_view = config.config_default_show
-    return render_title_template("user_edit.html", new_user=1, content=content, translations=translations,
+        content.locale = config.config_default_locale
+        content.default_language = config.config_default_language
+    return render_title_template("user_edit.html", new_user=1, content=content,
+                                 config=config, translations=translations,
                                  languages=languages, title=_(u"Add new user"), page="newuser",
                                  kobo_support=kobo_support, registered_oauth=oauth_check)
 
@@ -1560,7 +1614,7 @@ def edit_user(user_id):
     if not content or (not config.config_anonbrowse and content.name == "Guest"):
         flash(_(u"User not found"), category="error")
         return redirect(url_for('admin.admin'))
-    languages = calibre_db.speaking_language()
+    languages = calibre_db.speaking_language(return_all_languages=True)
     translations = babel.list_translations() + [LC('en')]
     kobo_support = feature_support['kobo'] and config.config_kobo_sync
     if request.method == "POST":
@@ -1573,6 +1627,7 @@ def edit_user(user_id):
                                  languages=languages,
                                  new_user=0,
                                  content=content,
+                                 config=config,
                                  registered_oauth=oauth_check,
                                  mail_configured=config.get_mail_server_configured(),
                                  kobo_support=kobo_support,
@@ -1580,7 +1635,7 @@ def edit_user(user_id):
                                  page="edituser")
 
 
-@admi.route("/admin/resetpassword/<int:user_id>")
+@admi.route("/admin/resetpassword/<int:user_id>", methods=["POST"])
 @login_required
 @admin_required
 def reset_user_password(user_id):
@@ -1682,7 +1737,8 @@ def get_updater_status():
                     "9": _(u'Update failed:') + u' ' + _(u'Connection error'),
                     "10": _(u'Update failed:') + u' ' + _(u'Timeout while establishing connection'),
                     "11": _(u'Update failed:') + u' ' + _(u'General error'),
-                    "12": _(u'Update failed:') + u' ' + _(u'Update File Could Not be Saved in Temp Dir')
+                    "12": _(u'Update failed:') + u' ' + _(u'Update file could not be saved in temp dir'),
+                    "13": _(u'Update failed:') + u' ' + _(u'Files could not be replaced during update')
                 }
                 status['text'] = text
                 updater_thread.status = 0
@@ -1736,6 +1792,8 @@ def ldap_import_create_user(user, user_data):
     content.password = ''  # dummy password which will be replaced by ldap one
     content.email = useremail
     content.kindle_mail = kindlemail
+    content.default_language = config.config_default_language
+    content.locale = config.config_default_locale
     content.role = config.config_default_role
     content.sidebar_view = config.config_default_show
     content.allowed_tags = config.config_allowed_tags
@@ -1753,7 +1811,7 @@ def ldap_import_create_user(user, user_data):
         return 0, message
 
 
-@admi.route('/import_ldap_users')
+@admi.route('/import_ldap_users', methods=["POST"])
 @login_required
 @admin_required
 def import_ldap_users():

cps/cache_buster.py

@@ -19,7 +19,6 @@
 # Inspired by https://github.com/ChrisTM/Flask-CacheBust
 # Uses query strings so CSS font files are found without having to resort to absolute URLs
 
-from __future__ import division, print_function, unicode_literals
 import os
 import hashlib
 

cps/cli.py

@@ -16,7 +16,6 @@
 #   You should have received a copy of the GNU General Public License
 #   along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import sys
 import os
 import argparse
@@ -46,21 +45,9 @@ parser.add_argument('-v', '--version', action='version', help='Shows version num
 parser.add_argument('-i', metavar='ip-address', help='Server IP-Address to listen')
 parser.add_argument('-s', metavar='user:pass', help='Sets specific username to new password')
 parser.add_argument('-f', action='store_true', help='Flag is depreciated and will be removed in next version')
+parser.add_argument('-l', action='store_true', help='Allow loading covers from localhost')
 args = parser.parse_args()
 
-if sys.version_info < (3, 0):
-    if args.p:
-        args.p = args.p.decode('utf-8')
-    if args.g:
-        args.g = args.g.decode('utf-8')
-    if args.k:
-        args.k = args.k.decode('utf-8')
-    if args.c:
-        args.c = args.c.decode('utf-8')
-    if args.s:
-        args.s = args.s.decode('utf-8')
-
-
 settingspath = args.p or os.path.join(_CONFIG_DIR, "app.db")
 gdpath       = args.g or os.path.join(_CONFIG_DIR, "gdrive.db")
 
@@ -91,6 +78,8 @@ if (args.k and not args.c) or (not args.k and args.c):
 if args.k == "":
     keyfilepath = ""
 
+# load covers from localhost
+allow_localhost = args.l or None
 # handle and check ip address argument
 ip_address = args.i or None
 if ip_address:

cps/comic.py

@@ -16,7 +16,6 @@
 #   You should have received a copy of the GNU General Public License
 #   along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import os
 
 from . import logger, isoLanguages
@@ -57,25 +56,25 @@ COVER_EXTENSIONS = ['.png', '.webp', '.bmp', '.jpg', '.jpeg']
 
 def _cover_processing(tmp_file_name, img, extension):
     tmp_cover_name = os.path.join(os.path.dirname(tmp_file_name), 'cover.jpg')
-    if use_IM:
-        # convert to jpg because calibre only supports jpg
     if extension in NO_JPEG_EXTENSIONS:
-            with Image(filename=tmp_file_name) as imgc:
+        if use_IM:
+            with Image(blob=img) as imgc:
                 imgc.format = 'jpeg'
                 imgc.transform_colorspace('rgb')
-                imgc.save(tmp_cover_name)
+                imgc.save(filename=tmp_cover_name)
                 return tmp_cover_name
-
+        else:
-    if not img:
             return None
-
+    if img:
         with open(tmp_cover_name, 'wb') as f:
             f.write(img)
         return tmp_cover_name
+    else:
+        return None
 
 
 def _extract_Cover_from_archive(original_file_extension, tmp_file_name, rarExecutable):
-    cover_data = None
+    cover_data = extension = None
     if original_file_extension.upper() == '.CBZ':
         cf = zipfile.ZipFile(tmp_file_name)
         for name in cf.namelist():
@@ -107,7 +106,7 @@ def _extract_Cover_from_archive(original_file_extension, tmp_file_name, rarExecu
                         break
         except Exception as ex:
             log.debug('Rarfile failed with error: %s', ex)
-    return cover_data
+    return cover_data, extension
 
 
 def _extractCover(tmp_file_name, original_file_extension, rarExecutable):
@@ -122,7 +121,7 @@ def _extractCover(tmp_file_name, original_file_extension, rarExecutable):
                     cover_data = archive.getPage(index)
                     break
     else:
-        cover_data = _extract_Cover_from_archive(original_file_extension, tmp_file_name, rarExecutable)
+        cover_data, extension = _extract_Cover_from_archive(original_file_extension, tmp_file_name, rarExecutable)
     return _cover_processing(tmp_file_name, cover_data, extension)
 
 

cps/config_sql.py

@@ -16,8 +16,6 @@
 #   You should have received a copy of the GNU General Public License
 #   along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-
-from __future__ import division, print_function, unicode_literals
 import os
 import sys
 import json
@@ -68,7 +66,7 @@ class _Settings(_Base):
     config_external_port = Column(Integer, default=constants.DEFAULT_PORT)
     config_certfile = Column(String)
     config_keyfile = Column(String)
-
+    config_trustedhosts = Column(String,default='')
     config_calibre_web_title = Column(String, default=u'Calibre-Web')
     config_books_per_page = Column(Integer, default=60)
     config_random_books = Column(Integer, default=4)
@@ -91,6 +89,8 @@ class _Settings(_Base):
 
     config_default_role = Column(SmallInteger, default=0)
     config_default_show = Column(SmallInteger, default=constants.ADMIN_USER_SIDEBAR)
+    config_default_language = Column(String(3), default="all")
+    config_default_locale = Column(String(2), default="en")
     config_columns_to_ignore = Column(String)
 
     config_denied_tags = Column(String, default="")
@@ -133,6 +133,7 @@ class _Settings(_Base):
     config_calibre = Column(String)
     config_rarfile_location = Column(String, default=None)
     config_upload_formats = Column(String, default=','.join(constants.EXTENSIONS_UPLOAD))
+    config_unicode_filename =Column(Boolean, default=False)
 
     config_updatechannel = Column(Integer, default=constants.UPDATE_STABLE)
 
@@ -360,10 +361,6 @@ def _migrate_table(session, orm_class):
                 session.query(column).first()
             except OperationalError as err:
                 log.debug("%s: %s", column_name, err.args[0])
-                if column.default is not None:
-                    if sys.version_info < (3, 0):
-                        if isinstance(column.default.arg, unicode):
-                            column.default.arg = column.default.arg.encode('utf-8')
                 if column.default is None:
                     column_default = ""
                 else:

cps/constants.py

@@ -16,7 +16,6 @@
 #   You should have received a copy of the GNU General Public License
 #   along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import sys
 import os
 from collections import namedtuple
@@ -31,12 +30,7 @@ HOME_CONFIG = os.path.isfile(os.path.join(os.path.dirname(os.path.abspath(__file
 UPDATER_AVAILABLE = True
 
 # Base dir is parent of current file, necessary if called from different folder
-if sys.version_info < (3, 0):
+BASE_DIR            = os.path.abspath(os.path.join(os.path.dirname(os.path.abspath(__file__)),os.pardir))
-    BASE_DIR            = os.path.abspath(os.path.join(
-        os.path.dirname(os.path.abspath(__file__)),os.pardir)).decode('utf-8')
-else:
-    BASE_DIR            = os.path.abspath(os.path.join(
-        os.path.dirname(os.path.abspath(__file__)),os.pardir))
 STATIC_DIR          = os.path.join(BASE_DIR, 'cps', 'static')
 TEMPLATES_DIR       = os.path.join(BASE_DIR, 'cps', 'templates')
 TRANSLATIONS_DIR    = os.path.join(BASE_DIR, 'cps', 'translations')
@@ -157,7 +151,7 @@ def selected_roles(dictionary):
 BookMeta = namedtuple('BookMeta', 'file_path, extension, title, author, cover, description, tags, series, '
                                   'series_id, languages, publisher')
 
-STABLE_VERSION = {'version': '0.6.13'}
+STABLE_VERSION = {'version': '0.6.16'}
 
 NIGHTLY_VERSION = {}
 NIGHTLY_VERSION[0] = '$Format:%H$'

cps/converter.py

@@ -16,7 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import os
 import re
 from flask_babel import gettext as _

cps/db.py

@@ -17,13 +17,13 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import sys
 import os
 import re
 import ast
 import json
 from datetime import datetime
+from urllib.parse import quote
 
 from sqlalchemy import create_engine
 from sqlalchemy import Table, Column, ForeignKey, CheckConstraint
@@ -167,6 +167,8 @@ class Identifiers(Base):
             return u"https://portal.issn.org/resource/ISSN/{0}".format(self.val)
         elif format_type == "isfdb":
             return u"http://www.isfdb.org/cgi-bin/pl.cgi?{0}".format(self.val)
+        elif self.val.lower().startswith("javascript:"):
+            return quote(self.val)
         else:
             return u"{0}".format(self.val)
 
@@ -175,8 +177,8 @@ class Comments(Base):
     __tablename__ = 'comments'
 
     id = Column(Integer, primary_key=True)
+    book = Column(Integer, ForeignKey('books.id'), nullable=False, unique=True)
     text = Column(String(collation='NOCASE'), nullable=False)
-    book = Column(Integer, ForeignKey('books.id'), nullable=False)
 
     def __init__(self, text, book):
         self.text = text
@@ -387,8 +389,6 @@ class Custom_Columns(Base):
 
     def get_display_dict(self):
         display_dict = ast.literal_eval(self.display)
-        if sys.version_info < (3, 0):
-            display_dict['enum_values'] = [x.decode('unicode_escape') for x in display_dict['enum_values']]
         return display_dict
 
 
@@ -398,7 +398,7 @@ class AlchemyEncoder(json.JSONEncoder):
         if isinstance(o.__class__, DeclarativeMeta):
             # an SQLAlchemy class
             fields = {}
-            for field in [x for x in dir(o) if not x.startswith('_') and x != 'metadata' and x!="password"]:
+            for field in [x for x in dir(o) if not x.startswith('_') and x != 'metadata' and x != "password"]:
                 if field == 'books':
                     continue
                 data = o.__getattribute__(field)
@@ -407,8 +407,11 @@ class AlchemyEncoder(json.JSONEncoder):
                         data = data.replace("'", "\'")
                     elif isinstance(data, InstrumentedList):
                         el = list()
+                        # ele = None
                         for ele in data:
-                            if ele.get:
+                            if hasattr(ele, 'value'):       # converter for custom_column values
+                                el.append(str(ele.value))
+                            elif ele.get:
                                 el.append(ele.get())
                             else:
                                 el.append(json.dumps(ele, cls=AlchemyEncoder))
@@ -550,11 +553,8 @@ class CalibreDB():
 
     @classmethod
     def setup_db(cls, config_calibre_dir, app_db_path):
-        # cls.config = config
         cls.dispose()
 
-        # toDo: if db changed -> delete shelfs, delete download books, delete read boks, kobo sync??
-
         if not config_calibre_dir:
             cls.config.invalidate()
             return False
@@ -612,7 +612,7 @@ class CalibreDB():
         return self.session.query(Data).filter(Data.book == book_id).filter(Data.format == file_format).first()
 
     # Language and content filters for displaying in the UI
-    def common_filters(self, allow_show_archived=False):
+    def common_filters(self, allow_show_archived=False, return_all_languages=False):
         if not allow_show_archived:
             archived_books = (
                 ub.session.query(ub.ArchivedBook)
@@ -625,10 +625,10 @@ class CalibreDB():
         else:
             archived_filter = true()
 
-        if current_user.filter_language() != "all":
+        if current_user.filter_language() == "all" or return_all_languages:
-            lang_filter = Books.languages.any(Languages.lang_code == current_user.filter_language())
-        else:
             lang_filter = true()
+        else:
+            lang_filter = Books.languages.any(Languages.lang_code == current_user.filter_language())
         negtags_list = current_user.list_denied_tags()
         postags_list = current_user.list_allowed_tags()
         neg_content_tags_filter = false() if negtags_list == [''] else Books.tags.any(Tags.name.in_(negtags_list))
@@ -740,6 +740,7 @@ class CalibreDB():
         self.session.connection().connection.connection.create_function("lower", 1, lcase)
         entries = self.session.query(database).filter(tag_filter). \
             filter(func.lower(database.name).ilike("%" + query + "%")).all()
+        # json_dumps = json.dumps([dict(name=escape(r.name.replace(*replace))) for r in entries])
         json_dumps = json.dumps([dict(name=r.name.replace(*replace)) for r in entries])
         return json_dumps
 
@@ -779,7 +780,7 @@ class CalibreDB():
 
     # read search results from calibre-database and return it (function is used for feed and simple search
     def get_search_results(self, term, offset=None, order=None, limit=None, *join):
-        order = order or [Books.sort]
+        order = order[0] if order else [Books.sort]
         pagination = None
         result = self.search_query(term, *join).order_by(*order).all()
         result_count = len(result)
@@ -795,22 +796,29 @@ class CalibreDB():
         return result[offset:limit_all], result_count, pagination
 
     # Creates for all stored languages a translated speaking name in the array for the UI
-    def speaking_language(self, languages=None):
+    def speaking_language(self, languages=None, return_all_languages=False, with_count=False, reverse_order=False):
         from . import get_locale
 
+        if with_count:
+            if not languages:
+                languages = self.session.query(Languages, func.count('books_languages_link.book'))\
+                    .join(books_languages_link).join(Books)\
+                    .filter(self.common_filters(return_all_languages=return_all_languages)) \
+                    .group_by(text('books_languages_link.lang_code')).all()
+            for lang in languages:
+                lang[0].name = isoLanguages.get_language_name(get_locale(), lang[0].lang_code)
+            return sorted(languages, key=lambda x: x[0].name, reverse=reverse_order)
+        else:
             if not languages:
                 languages = self.session.query(Languages) \
                     .join(books_languages_link) \
                     .join(Books) \
-                .filter(self.common_filters()) \
+                    .filter(self.common_filters(return_all_languages=return_all_languages)) \
                     .group_by(text('books_languages_link.lang_code')).all()
             for lang in languages:
-            try:
+                lang.name = isoLanguages.get_language_name(get_locale(), lang.lang_code)
-                cur_l = LC.parse(lang.lang_code)
+            return sorted(languages, key=lambda x: x.name, reverse=reverse_order)
-                lang.name = cur_l.get_language_name(get_locale())
+
-            except UnknownLocaleError:
-                lang.name = _(isoLanguages.get(part3=lang.lang_code).name)
-        return languages
 
     def update_title_sort(self, config, conn=None):
         # user defined sort function for calibre databases (Series, etc.)

cps/dep_check.py

@@ -0,0 +1,96 @@
+import os
+import re
+
+from .constants import BASE_DIR
+try:
+    from importlib_metadata import version
+    importlib = True
+    ImportNotFound = BaseException
+except ImportError:
+    importlib = False
+
+
+if not importlib:
+    try:
+        import pkg_resources
+        from pkg_resources import DistributionNotFound as ImportNotFound
+        pkgresources = True
+    except ImportError as e:
+        pkgresources = False
+
+def load_dependencys(optional=False):
+    deps = list()
+    if importlib or pkgresources:
+        if optional:
+            req_path = os.path.join(BASE_DIR, "optional-requirements.txt")
+        else:
+            req_path = os.path.join(BASE_DIR, "requirements.txt")
+        if os.path.exists(req_path):
+            with open(req_path, 'r') as f:
+                for line in f:
+                    if not line.startswith('#') and not line == '\n' and not line.startswith('git'):
+                        res = re.match(r'(.*?)([<=>\s]+)([\d\.]+),?\s?([<=>\s]+)?([\d\.]+)?', line.strip())
+                        try:
+                            if importlib:
+                                dep_version = version(res.group(1))
+                            else:
+                                dep_version = pkg_resources.get_distribution(res.group(1)).version
+                        except ImportNotFound:
+                            if optional:
+                                continue
+                            dep_version = "not installed"
+                        deps.append([dep_version, res.group(1), res.group(2), res.group(3), res.group(4), res.group(5)])
+    return deps
+
+
+def dependency_check(optional=False):
+    d = list()
+    deps = load_dependencys(optional)
+    for dep in deps:
+        try:
+            dep_version_int = [int(x) for x in dep[0].split('.')]
+            low_check = [int(x) for x in dep[3].split('.')]
+            high_check = [int(x) for x in dep[5].split('.')]
+        except AttributeError:
+            high_check = None
+        except ValueError:
+            d.append({'name': dep[1],
+                     'target': "available",
+                     'found': "Not available"
+                     })
+            continue
+
+        if dep[2].strip() == "==":
+            if dep_version_int != low_check:
+                d.append({'name': dep[1],
+                            'found': dep[0],
+                            "target": dep[2] + dep[3]})
+                continue
+        elif dep[2].strip() == ">=":
+            if dep_version_int < low_check:
+                d.append({'name': dep[1],
+                            'found': dep[0],
+                            "target": dep[2] + dep[3]})
+                continue
+        elif dep[2].strip() == ">":
+            if dep_version_int <= low_check:
+                d.append({'name': dep[1],
+                            'found': dep[0],
+                            "target": dep[2] + dep[3]})
+                continue
+        if dep[4] and dep[5]:
+            if dep[4].strip() == "<":
+                if dep_version_int >= high_check:
+                    d.append(
+                        {'name': dep[1],
+                         'found': dep[0],
+                         "target": dep[4] + dep[5]})
+                    continue
+            elif dep[4].strip() == "<=":
+                if dep_version_int > high_check:
+                    d.append(
+                        {'name': dep[1],
+                         'found': dep[0],
+                         "target": dep[4] + dep[5]})
+                    continue
+    return d

cps/editbooks.py

@@ -20,19 +20,19 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import os
 from datetime import datetime
 import json
 from shutil import copyfile
 from uuid import uuid4
 from markupsafe import escape
+from functools import wraps
+
 try:
     from lxml.html.clean import clean_html
 except ImportError:
     pass
 
-
 # Improve this to check if scholarly is available in a global way, like other pythonic libraries
 try:
     from scholarly import scholarly
@@ -40,14 +40,12 @@ try:
 except ImportError:
     have_scholar = False
 
-from babel import Locale as LC
-from babel.core import UnknownLocaleError
 from flask import Blueprint, request, flash, redirect, url_for, abort, Markup, Response
 from flask_babel import gettext as _
 from flask_login import current_user, login_required
 from sqlalchemy.exc import OperationalError, IntegrityError
 from sqlite3 import OperationalError as sqliteOperationalError
-from . import constants, logger, isoLanguages, gdriveutils, uploader, helper
+from . import constants, logger, isoLanguages, gdriveutils, uploader, helper, kobo_sync_status
 from . import config, get_locale, ub, db
 from . import calibre_db
 from .services.worker import WorkerThread
@@ -55,13 +53,6 @@ from .tasks.upload import TaskUpload
 from .render_template import render_title_template
 from .usermanagement import login_required_if_no_ano
 
-try:
-    from functools import wraps
-except ImportError:
-    pass  # We're not using Python 3
-
-
-
 
 editbook = Blueprint('editbook', __name__)
 log = logger.create()
@@ -70,7 +61,7 @@ log = logger.create()
 def upload_required(f):
     @wraps(f)
     def inner(*args, **kwargs):
-        if current_user.role_upload() or current_user.role_admin():
+        if current_user.role_upload():
             return f(*args, **kwargs)
         abort(403)
 
@@ -241,17 +232,17 @@ def modify_identifiers(input_identifiers, db_identifiers, db_session):
             changed = True
     return changed, error
 
-@editbook.route("/ajax/delete/<int:book_id>")
+@editbook.route("/ajax/delete/<int:book_id>", methods=["POST"])
 @login_required
 def delete_book_from_details(book_id):
-    return Response(delete_book(book_id, "", True), mimetype='application/json')
+    return Response(delete_book_from_table(book_id, "", True), mimetype='application/json')
 
 
-@editbook.route("/delete/<int:book_id>", defaults={'book_format': ""})
+@editbook.route("/delete/<int:book_id>", defaults={'book_format': ""}, methods=["POST"])
-@editbook.route("/delete/<int:book_id>/<string:book_format>")
+@editbook.route("/delete/<int:book_id>/<string:book_format>", methods=["POST"])
 @login_required
 def delete_book_ajax(book_id, book_format):
-    return delete_book(book_id, book_format, False)
+    return delete_book_from_table(book_id, book_format, False)
 
 
 def delete_whole_book(book_id, book):
@@ -321,7 +312,7 @@ def render_delete_book_result(book_format, jsonResponse, warning, book_id):
             return redirect(url_for('web.index'))
 
 
-def delete_book(book_id, book_format, jsonResponse):
+def delete_book_from_table(book_id, book_format, jsonResponse):
     warning = {}
     if current_user.role_delete_books():
         book = calibre_db.get_book(book_id)
@@ -350,6 +341,8 @@ def delete_book(book_id, book_format, jsonResponse):
                 else:
                     calibre_db.session.query(db.Data).filter(db.Data.book == book.id).\
                         filter(db.Data.format == book_format).delete()
+                    if book_format.upper() in ['KEPUB', 'EPUB', 'EPUB3']:
+                        kobo_sync_status.remove_synced_book(book.id, True)
                 calibre_db.session.commit()
             except Exception as ex:
                 log.debug_or_exception(ex)
@@ -367,6 +360,15 @@ def delete_book(book_id, book_format, jsonResponse):
             # book not found
             log.error('Book with id "%s" could not be deleted: not found', book_id)
         return render_delete_book_result(book_format, jsonResponse, warning, book_id)
+    message = _("You are missing permissions to delete books")
+    if jsonResponse:
+        return json.dumps({"location": url_for("editbook.edit_book", book_id=book_id),
+                           "type": "danger",
+                           "format": "",
+                           "message": message})
+    else:
+        flash(message, category="error")
+        return redirect(url_for('editbook.edit_book', book_id=book_id))
 
 
 def render_edit_book(book_id):
@@ -457,11 +459,12 @@ def edit_book_series_index(series_index, book):
     if not series_index.replace('.', '', 1).isdigit():
         flash(_("%(seriesindex)s is not a valid number, skipping", seriesindex=series_index), category="warning")
         return False
-    if book.series_index != series_index:
+    if str(book.series_index) != series_index:
         book.series_index = series_index
         modif_date = True
     return modif_date
 
+
 # Handle book comments/description
 def edit_book_comments(comments, book):
     modif_date = False
@@ -486,11 +489,11 @@ def edit_book_languages(languages, book, upload=False, invalid=None):
     else:
         input_l = isoLanguages.get_valid_language_codes(get_locale(), input_languages, unknown_languages)
     for l in unknown_languages:
-        log.error('%s is not a valid language', l)
+        log.error("'%s' is not a valid language", l)
         if isinstance(invalid, list):
             invalid.append(l)
         else:
-            flash(_(u"%(langname)s is not a valid language", langname=l), category="warning")
+            raise ValueError(_(u"'%(langname)s' is not a valid language", langname=l))
     # ToDo: Not working correct
     if upload and len(input_l) == 1:
         # If the language of the file is excluded from the users view, it's not imported, to allow the user to view
@@ -576,10 +579,19 @@ def edit_cc_data_string(book, c, to_save, cc_db_value, cc_string):
         getattr(book, cc_string).append(new_cc)
     return changed, to_save
 
+def edit_single_cc_data(book_id, book, column_id, to_save):
+    cc = (calibre_db.session.query(db.Custom_Columns)
+          .filter(db.Custom_Columns.datatype.notin_(db.cc_exceptions))
+          .filter(db.Custom_Columns.id == column_id)
+          .all())
+    return edit_cc_data(book_id, book, to_save, cc)
 
-def edit_cc_data(book_id, book, to_save):
+def edit_all_cc_data(book_id, book, to_save):
-    changed = False
     cc = calibre_db.session.query(db.Custom_Columns).filter(db.Custom_Columns.datatype.notin_(db.cc_exceptions)).all()
+    return edit_cc_data(book_id, book, to_save, cc)
+
+def edit_cc_data(book_id, book, to_save, cc):
+    changed = False
     for c in cc:
         cc_string = "custom_column_" + str(c.id)
         if not c.is_multiple:
@@ -814,7 +826,7 @@ def edit_book(book_id):
             # handle book ratings
             modif_date |= edit_book_ratings(to_save, book)
             # handle cc data
-            modif_date |= edit_cc_data(book_id, book, to_save)
+            modif_date |= edit_all_cc_data(book_id, book, to_save)
 
             if to_save["pubdate"]:
                 try:
@@ -826,6 +838,8 @@ def edit_book(book_id):
 
             if modif_date:
                 book.last_modified = datetime.utcnow()
+                kobo_sync_status.remove_synced_book(edited_books_id, all=True)
+
             calibre_db.session.merge(book)
             calibre_db.session.commit()
             if config.config_use_google_drive:
@@ -839,6 +853,10 @@ def edit_book(book_id):
             calibre_db.session.rollback()
             flash(error, category="error")
             return render_edit_book(book_id)
+    except ValueError as e:
+        calibre_db.session.rollback()
+        flash(str(e), category="error")
+        return redirect(url_for('web.show_book', book_id=book.id))
     except Exception as ex:
         log.debug_or_exception(ex)
         calibre_db.session.rollback()
@@ -935,7 +953,11 @@ def create_book_on_upload(modif_date, meta):
     modif_date |= edit_book_series_index(meta.series_id, db_book)
 
     # add languages
-    modif_date |= edit_book_languages(meta.languages, db_book, upload=True)
+    invalid=[]
+    modif_date |= edit_book_languages(meta.languages, db_book, upload=True, invalid=invalid)
+    if invalid:
+        for l in invalid:
+            flash(_(u"'%(langname)s' is not a valid language", langname=l), category="warning")
 
     # handle tags
     modif_date |= edit_book_tags(meta.tags, db_book)
@@ -998,7 +1020,7 @@ def move_coverfile(meta, db_book):
               category="error")
 
 
-@editbook.route("/upload", methods=["GET", "POST"])
+@editbook.route("/upload", methods=["POST"])
 @login_required_if_no_ano
 @upload_required
 def upload():
@@ -1080,23 +1102,18 @@ def convert_bookformat(book_id):
         flash(_(u"There was an error converting this book: %(res)s", res=rtn), category="error")
     return redirect(url_for('editbook.edit_book', book_id=book_id))
 
-@editbook.route("/scholarsearch/<query>",methods=['GET'])
+@editbook.route("/ajax/getcustomenum/<int:c_id>")
-@login_required_if_no_ano
+@login_required
-@edit_required
+def table_get_custom_enum(c_id):
-def scholar_search(query):
+    ret = list()
-    if have_scholar:
+    cc = (calibre_db.session.query(db.Custom_Columns)
-        scholar_gen = scholarly.search_pubs(' '.join(query.split('+')))
+              .filter(db.Custom_Columns.id == c_id)
-        i=0
+              .filter(db.Custom_Columns.datatype.notin_(db.cc_exceptions)).one_or_none())
-        result = []
+    ret.append({'value': "", 'text': ""})
-        for publication in scholar_gen:
+    for idx, en in enumerate(cc.get_display_dict()['enum_values']):
-            del publication['source']
+        ret.append({'value': en, 'text': en})
-            result.append(publication)
+    return json.dumps(ret)
-            i+=1
+
-            if(i>=10):
-                break
-        return Response(json.dumps(result),mimetype='application/json')
-    else:
-        return "[]"
 
 @editbook.route("/ajax/editbooks/<param>", methods=['POST'])
 @login_required_if_no_ano
@@ -1131,10 +1148,7 @@ def edit_list_book(param):
         else:
             lang_names = list()
             for lang in book.languages:
-                try:
+                lang_names.append(isoLanguages.get_language_name(get_locale(), lang.lang_code))
-                    lang_names.append(LC.parse(lang.lang_code).get_language_name(get_locale()))
-                except UnknownLocaleError:
-                    lang_names.append(_(isoLanguages.get(part3=lang.lang_code).name))
             ret =  Response(json.dumps({'success': True, 'newValue':  ', '.join(lang_names)}),
                             mimetype='application/json')
     elif param =='author_sort':
@@ -1151,12 +1165,23 @@ def edit_list_book(param):
         book.sort = vals['value']
         ret = Response(json.dumps({'success': True, 'newValue':  book.sort}),
                        mimetype='application/json')
+    elif param =='comments':
+        edit_book_comments(vals['value'], book)
+        ret = Response(json.dumps({'success': True, 'newValue':  book.comments[0].text}),
+                       mimetype='application/json')
     elif param =='authors':
         input_authors, __ = handle_author_on_edit(book, vals['value'], vals.get('checkA', None) == "true")
         helper.update_dir_stucture(book.id, config.config_calibre_dir, input_authors[0])
         ret = Response(json.dumps({'success': True,
                                    'newValue':  ' & '.join([author.replace('|',',') for author in input_authors])}),
                        mimetype='application/json')
+    elif param.startswith("custom_column_"):
+        new_val = dict()
+        new_val[param] = vals['value']
+        edit_single_cc_data(book.id, book, param[14:], new_val)
+        ret = Response(json.dumps({'success': True, 'newValue': vals['value']}),
+                       mimetype='application/json')
+
     book.last_modified = datetime.utcnow()
     try:
         calibre_db.session.commit()
@@ -1235,7 +1260,7 @@ def merge_list_book():
                                                         element.format,
                                                         element.uncompressed_size,
                                                         to_name))
-                    delete_book(from_book.id,"", True)
+                    delete_book_from_table(from_book.id,"", True)
                     return json.dumps({'success': True})
     return ""
 

cps/epub.py

@@ -16,7 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import os
 import zipfile
 from lxml import etree

cps/fb2.py

@@ -16,7 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 from lxml import etree
 
 from .constants import BookMeta

cps/gdrive.py

@@ -20,9 +20,7 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import os
-import sys
 import hashlib
 import json
 import tempfile
@@ -34,7 +32,7 @@ from flask import Blueprint, flash, request, redirect, url_for, abort
 from flask_babel import gettext as _
 from flask_login import login_required
 
-from . import logger, gdriveutils, config, ub, calibre_db
+from . import logger, gdriveutils, config, ub, calibre_db, csrf
 from .admin import admin_required
 
 gdrive = Blueprint('gdrive', __name__, url_prefix='/gdrive')
@@ -111,15 +109,16 @@ def revoke_watch_gdrive():
         try:
             gdriveutils.stopChannel(gdriveutils.Gdrive.Instance().drive, last_watch_response['id'],
                                     last_watch_response['resourceId'])
-        except HttpError:
+        except (HttpError, AttributeError):
             pass
         config.config_google_drive_watch_changes_response = {}
         config.save()
     return redirect(url_for('admin.db_configuration'))
 
-
+try:
-@gdrive.route("/watch/callback", methods=['GET', 'POST'])
+    @csrf.exempt
-def on_received_watch_confirmation():
+    @gdrive.route("/watch/callback", methods=['GET', 'POST'])
+    def on_received_watch_confirmation():
         if not config.config_google_drive_watch_changes_response:
             return ''
         if request.headers.get('X-Goog-Channel-Token') != gdrive_watch_callback_token \
@@ -137,9 +136,6 @@ def on_received_watch_confirmation():
             response = gdriveutils.getChangeById(gdriveutils.Gdrive.Instance().drive, j['id'])
             log.debug('%r', response)
             if response:
-            if sys.version_info < (3, 0):
-                dbpath = os.path.join(config.config_calibre_dir, "metadata.db")
-            else:
                 dbpath = os.path.join(config.config_calibre_dir, "metadata.db").encode()
                 if not response['deleted'] and response['file']['title'] == 'metadata.db' \
                     and response['file']['md5Checksum'] != hashlib.md5(dbpath):  # nosec
@@ -158,3 +154,5 @@ def on_received_watch_confirmation():
         except Exception as ex:
             log.debug_or_exception(ex)
         return ''
+except AttributeError:
+    pass

cps/gdriveutils.py

@@ -16,7 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import os
 import json
 import shutil
@@ -36,6 +35,15 @@ except ImportError:
 from sqlalchemy.exc import OperationalError, InvalidRequestError
 from sqlalchemy.sql.expression import text
 
+try:
+    from six import __version__ as six_version
+except ImportError:
+    six_version = "not installed"
+try:
+    from httplib2 import __version__ as httplib2_version
+except ImportError:
+    httplib2_version = "not installed"
+
 try:
     from apiclient import errors
     from httplib2 import ServerNotFoundError
@@ -48,11 +56,13 @@ try:
     from pydrive2.auth import GoogleAuth
     from pydrive2.drive import GoogleDrive
     from pydrive2.auth import RefreshError
+    from pydrive2.files import ApiRequestError
 except ImportError as err:
     try:
         from pydrive.auth import GoogleAuth
         from pydrive.drive import GoogleDrive
         from pydrive.auth import RefreshError
+        from pydrive.files import ApiRequestError
     except ImportError as err:
         importError = err
         gdrive_support = False
@@ -314,6 +324,11 @@ def getFolderId(path, drive):
         log.error("gdrive.db DB is not Writeable")
         log.debug('Database error: %s', ex)
         session.rollback()
+    except ApiRequestError as ex:
+        log.error('{} {}'.format(ex.error['message'], path))
+        session.rollback()
+    except RefreshError as ex:
+        log.error(ex)
     return currentFolderId
 
 
@@ -660,3 +675,8 @@ def get_error_text(client_secrets=None):
         return 'Callback url (redirect url) is missing in client_secrets.json'
     if client_secrets:
         client_secrets.update(filedata['web'])
+
+
+def get_versions():
+    return {'six': six_version,
+            'httplib2': httplib2_version}

cps/helper.py

@@ -17,19 +17,18 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
-import sys
 import os
 import io
 import mimetypes
 import re
 import shutil
-import time
+import socket
 import unicodedata
 from datetime import datetime, timedelta
 from tempfile import gettempdir
-
+from urllib.parse import urlparse
 import requests
+
 from babel.dates import format_datetime
 from babel.units import format_unit
 from flask import send_from_directory, make_response, redirect, abort, url_for
@@ -39,11 +38,7 @@ from sqlalchemy.sql.expression import true, false, and_, text, func
 from werkzeug.datastructures import Headers
 from werkzeug.security import generate_password_hash
 from markupsafe import escape
-
+from urllib.parse import quote
-try:
-    from urllib.parse import quote
-except ImportError:
-    from urllib import quote
 
 try:
     import unidecode
@@ -51,7 +46,7 @@ try:
 except ImportError:
     use_unidecode = False
 
-from . import calibre_db
+from . import calibre_db, cli
 from .tasks.convert import TaskConvert
 from . import logger, config, get_locale, db, ub
 from . import gdriveutils as gd
@@ -64,7 +59,7 @@ log = logger.create()
 
 try:
     from wand.image import Image
-    from wand.exceptions import MissingDelegateError
+    from wand.exceptions import MissingDelegateError, BlobError
     use_IM = True
 except (ImportError, RuntimeError) as e:
     log.debug('Cannot import Image, generating covers from non jpg files will not work: %s', e)
@@ -234,16 +229,14 @@ def get_valid_filename(value, replace_whitespace=True):
         value = value[:-1]+u'_'
     value = value.replace("/", "_").replace(":", "_").strip('\0')
     if use_unidecode:
+        if config.config_unicode_filename:
             value = (unidecode.unidecode(value))
     else:
         value = value.replace(u'§', u'SS')
         value = value.replace(u'ß', u'ss')
         value = unicodedata.normalize('NFKD', value)
         re_slugify = re.compile(r'[\W\s-]', re.UNICODE)
-        if isinstance(value, str):  # Python3 str, Python2 unicode
         value = re_slugify.sub('', value)
-        else:
-            value = unicode(re_slugify.sub('', value))
     if replace_whitespace:
         #  *+:\"/<>? are replaced by _
         value = re.sub(r'[*+:\\\"/<>?]+', u'_', value, flags=re.U)
@@ -252,10 +245,7 @@ def get_valid_filename(value, replace_whitespace=True):
     value = value[:128].strip()
     if not value:
         raise ValueError("Filename cannot be empty")
-    if sys.version_info.major == 3:
     return value
-    else:
-        return value.decode('utf-8')
 
 
 def split_authors(values):
@@ -495,9 +485,6 @@ def reset_password(user_id):
 def generate_random_password():
     s = "abcdefghijklmnopqrstuvwxyz01234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ!@#$%&*()?"
     passlen = 8
-    if sys.version_info < (3, 0):
-        return "".join(s[ord(c) % len(s)] for c in os.urandom(passlen))
-    else:
     return "".join(s[c % len(s)] for c in os.urandom(passlen))
 
 
@@ -597,10 +584,17 @@ def get_book_cover_internal(book, use_generic_cover_on_failure):
 # saves book cover from url
 def save_cover_from_url(url, book_path):
     try:
+        if not cli.allow_localhost:
+            # 127.0.x.x, localhost, [::1], [::ffff:7f00:1]
+            ip = socket.getaddrinfo(urlparse(url).hostname, 0)[0][4][0]
+            if ip.startswith("127.") or ip.startswith('::ffff:7f') or ip == "::1":
+                log.error("Localhost was accessed for cover upload")
+                return False, _("You are not allowed to access localhost for cover uploads")
         img = requests.get(url, timeout=(10, 200))      # ToDo: Error Handling
         img.raise_for_status()
         return save_cover(img, book_path)
-    except (requests.exceptions.HTTPError,
+    except (socket.gaierror,
+            requests.exceptions.HTTPError,
             requests.exceptions.ConnectionError,
             requests.exceptions.Timeout) as ex:
         log.info(u'Cover Download Error %s', ex)
@@ -647,6 +641,7 @@ def save_cover(img, book_path):
             return False, _("Only jpg/jpeg/png/webp/bmp files are supported as coverfile")
         # convert to jpg because calibre only supports jpg
         if content_type != 'image/jpg':
+            try:
                 if hasattr(img, 'stream'):
                     imgc = Image(blob=img.stream)
                 else:
@@ -654,6 +649,9 @@ def save_cover(img, book_path):
                 imgc.format = 'jpeg'
                 imgc.transform_colorspace("rgb")
                 img = imgc
+            except (BlobError, MissingDelegateError):
+                log.error("Invalid cover file content")
+                return False, _("Invalid cover file content")
     else:
         if content_type not in 'image/jpeg':
             log.error("Only jpg/jpeg files are supported as coverfile")
@@ -678,9 +676,9 @@ def save_cover(img, book_path):
 
 def do_download_file(book, book_format, client, data, headers):
     if config.config_use_google_drive:
-        startTime = time.time()
+        #startTime = time.time()
         df = gd.getFileFromEbooksFolder(book.path, data.name + "." + book_format)
-        log.debug('%s', time.time() - startTime)
+        #log.debug('%s', time.time() - startTime)
         if df:
             return gd.do_gdrive_download(df, headers)
         else:
@@ -712,8 +710,6 @@ def check_unrar(unrarLocation):
         return _('Unrar binary file not found')
 
     try:
-        if sys.version_info < (3, 0):
-            unrarLocation = unrarLocation.encode(sys.getfilesystemencoding())
         unrarLocation = [unrarLocation]
         value = process_wait(unrarLocation, pattern='UNRAR (.*) freeware')
         if value:
@@ -830,7 +826,7 @@ def get_cc_columns(filter_config_custom_read=False):
 
 def get_download_link(book_id, book_format, client):
     book_format = book_format.split(".")[0]
-    book = calibre_db.get_filtered_book(book_id)
+    book = calibre_db.get_filtered_book(book_id, allow_show_archived=True)
     if book:
         data1 = calibre_db.get_book_format(book.id, book_format.upper())
     else:
@@ -842,8 +838,8 @@ def get_download_link(book_id, book_format, client):
             ub.update_download(book_id, int(current_user.id))
         file_name = book.title
         if len(book.authors) > 0:
-            file_name = book.authors[0].name + '_' + file_name
+            file_name = file_name + ' - ' + book.authors[0].name
-        file_name = get_valid_filename(file_name)
+        file_name = get_valid_filename(file_name, replace_whitespace=False)
         headers = Headers()
         headers["Content-Type"] = mimetypes.types_map.get('.' + book_format, "application/octet-stream")
         headers["Content-Disposition"] = "attachment; filename=%s.%s; filename*=UTF-8''%s.%s" % (

cps/isoLanguages.py

@@ -16,9 +16,10 @@
 #   You should have received a copy of the GNU General Public License
 #   along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
-
 from .iso_language_names import LANGUAGE_NAMES as _LANGUAGE_NAMES
+from . import logger
+
+log = logger.create()
 
 
 try:
@@ -52,7 +53,11 @@ def get_language_names(locale):
 
 
 def get_language_name(locale, lang_code):
+    try:
         return get_language_names(locale)[lang_code]
+    except KeyError:
+        log.error('Missing translation for language name: {}'.format(lang_code))
+        return "Unknown"
 
 
 def get_language_codes(locale, language_names, remainder=None):

cps/jinjia.py

@@ -22,7 +22,6 @@
 
 # custom jinja filters
 
-from __future__ import division, print_function, unicode_literals
 import datetime
 import mimetypes
 from uuid import uuid4

cps/kobo.py

@@ -19,15 +19,11 @@
 
 import base64
 import datetime
-import sys
 import os
 import uuid
 from time import gmtime, strftime
-
+import json
-try:
+from urllib.parse import unquote
-    from urllib import unquote
-except ImportError:
-    from urllib.parse import unquote
 
 from flask import (
     Blueprint,
@@ -47,7 +43,8 @@ from sqlalchemy.exc import StatementError
 from sqlalchemy.sql import select
 import requests
 
-from . import config, logger, kobo_auth, db, calibre_db, helper, shelf as shelf_lib, ub
+
+from . import config, logger, kobo_auth, db, calibre_db, helper, shelf as shelf_lib, ub, csrf, kobo_sync_status
 from .constants import sqlalchemy_version2
 from .helper import get_download_link
 from .services import SyncToken as SyncToken
@@ -102,6 +99,8 @@ def make_request_to_kobo_store(sync_token=None):
         allow_redirects=False,
         timeout=(2, 10)
     )
+    log.debug("Content: " + str(store_response.content))
+    log.debug("StatusCode: " + str(store_response.status_code))
     return store_response
 
 
@@ -110,7 +109,8 @@ def redirect_or_proxy_request():
         if request.method == "GET":
             return redirect(get_store_url_for_current_request(), 307)
         else:
-            # The Kobo device turns other request types into GET requests on redirects, so we instead proxy to the Kobo store ourselves.
+            # The Kobo device turns other request types into GET requests on redirects,
+            # so we instead proxy to the Kobo store ourselves.
             store_response = make_request_to_kobo_store()
 
             response_headers = store_response.headers
@@ -142,12 +142,16 @@ def HandleSyncRequest():
     if not current_app.wsgi_app.is_proxied:
         log.debug('Kobo: Received unproxied request, changed request port to external server port')
 
-    # TODO: Limit the number of books return per sync call, and rely on the sync-continuatation header
+    # if no books synced don't respect sync_token
-    # instead so that the device triggers another sync.
+    if not ub.session.query(ub.KoboSyncedBooks).filter(ub.KoboSyncedBooks.user_id == current_user.id).count():
+        sync_token.books_last_modified = datetime.datetime.min
+        sync_token.books_last_created = datetime.datetime.min
+        sync_token.reading_state_last_modified = datetime.datetime.min
 
-    new_books_last_modified = sync_token.books_last_modified
+    new_books_last_modified = sync_token.books_last_modified # needed for sync selected shelfs only
-    new_books_last_created = sync_token.books_last_created
+    new_books_last_created = sync_token.books_last_created # needed to distinguish between new and changed entitlement
     new_reading_state_last_modified = sync_token.reading_state_last_modified
+
     new_archived_last_modified = datetime.datetime.min
     sync_results = []
 
@@ -169,10 +173,13 @@ def HandleSyncRequest():
                                                        ub.BookShelf.date_added,
                                                        ub.ArchivedBook.is_archived)
         changed_entries = (changed_entries
-                .join(db.Data).outerjoin(ub.ArchivedBook, db.Books.id == ub.ArchivedBook.book_id)
+                           .join(db.Data).outerjoin(ub.ArchivedBook, and_(db.Books.id == ub.ArchivedBook.book_id,
-                .filter(or_(db.Books.last_modified > sync_token.books_last_modified,
+                                                                          ub.ArchivedBook.user_id == current_user.id))
-                            ub.BookShelf.date_added > sync_token.books_last_modified))
+                           .filter(db.Books.id.notin_(calibre_db.session.query(ub.KoboSyncedBooks.book_id)
-                .filter(db.Data.format.in_(KOBO_FORMATS)).filter(calibre_db.common_filters())
+                                           .filter(ub.KoboSyncedBooks.user_id == current_user.id)))
+                .filter(ub.BookShelf.date_added > sync_token.books_last_modified)
+                .filter(db.Data.format.in_(KOBO_FORMATS))
+                .filter(calibre_db.common_filters(allow_show_archived=True))
                 .order_by(db.Books.id)
                 .order_by(ub.ArchivedBook.last_modified)
                 .join(ub.BookShelf, db.Books.id == ub.BookShelf.book_id)
@@ -189,22 +196,23 @@ def HandleSyncRequest():
                                                        ub.ArchivedBook.last_modified,
                                                        ub.ArchivedBook.is_archived)
         changed_entries = (changed_entries
-                .join(db.Data).outerjoin(ub.ArchivedBook, db.Books.id == ub.ArchivedBook.book_id)
+                   .join(db.Data).outerjoin(ub.ArchivedBook, and_(db.Books.id == ub.ArchivedBook.book_id,
-                .filter(db.Books.last_modified > sync_token.books_last_modified)
+                                                                  ub.ArchivedBook.user_id == current_user.id))
-                .filter(calibre_db.common_filters())
+                   .filter(db.Books.id.notin_(calibre_db.session.query(ub.KoboSyncedBooks.book_id)
+                                              .filter(ub.KoboSyncedBooks.user_id == current_user.id)))
+                   .filter(calibre_db.common_filters(allow_show_archived=True))
                    .filter(db.Data.format.in_(KOBO_FORMATS))
                    .order_by(db.Books.last_modified)
                    .order_by(db.Books.id)
         )
 
-    if sync_token.books_last_id > -1:
-        changed_entries = changed_entries.filter(db.Books.id > sync_token.books_last_id)
 
     reading_states_in_new_entitlements = []
     if sqlalchemy_version2:
         books = calibre_db.session.execute(changed_entries.limit(SYNC_ITEM_LIMIT))
     else:
         books = changed_entries.limit(SYNC_ITEM_LIMIT)
+    log.debug("Books to Sync: {}".format(len(books.all())))
     for book in books:
         formats = [data.format for data in book.Books.data]
         if not 'KEPUB' in formats and config.config_kepubifypath and 'EPUB' in formats:
@@ -244,14 +252,17 @@ def HandleSyncRequest():
             pass
 
         new_books_last_created = max(ts_created, new_books_last_created)
+        kobo_sync_status.add_synced_books(book.Books.id)
 
     if sqlalchemy_version2:
         max_change = calibre_db.session.execute(changed_entries
                                                 .filter(ub.ArchivedBook.is_archived)
+                                                .filter(ub.ArchivedBook.user_id == current_user.id)
                                                 .order_by(func.datetime(ub.ArchivedBook.last_modified).desc()))\
             .columns(db.Books).first()
     else:
-        max_change = changed_entries.from_self().filter(ub.ArchivedBook.is_archived) \
+        max_change = changed_entries.from_self().filter(ub.ArchivedBook.is_archived)\
+            .filter(ub.ArchivedBook.user_id==current_user.id) \
             .order_by(func.datetime(ub.ArchivedBook.last_modified).desc()).first()
 
     max_change = max_change.last_modified if max_change else new_archived_last_modified
@@ -263,11 +274,10 @@ def HandleSyncRequest():
         entries = calibre_db.session.execute(changed_entries).all()
         book_count = len(entries)
     else:
-        entries = changed_entries.all()
         book_count = changed_entries.count()
     # last entry:
-    books_last_id = entries[-1].Books.id or -1 if book_count else -1
+    cont_sync = bool(book_count)
-
+    log.debug("Remaining books to Sync: {}".format(book_count))
     # generate reading state data
     changed_reading_states = ub.session.query(ub.KoboReadingState)
 
@@ -278,18 +288,19 @@ def HandleSyncRequest():
             .filter(current_user.id == ub.Shelf.user_id)\
             .filter(ub.Shelf.kobo_sync,
                     or_(
-                        func.datetime(ub.KoboReadingState.last_modified) > sync_token.reading_state_last_modified,
+                        ub.KoboReadingState.last_modified > sync_token.reading_state_last_modified,
                         func.datetime(ub.BookShelf.date_added) > sync_token.books_last_modified
                     )).distinct()
     else:
         changed_reading_states = changed_reading_states.filter(
-            func.datetime(ub.KoboReadingState.last_modified) > sync_token.reading_state_last_modified)
+            ub.KoboReadingState.last_modified > sync_token.reading_state_last_modified)
 
     changed_reading_states = changed_reading_states.filter(
         and_(ub.KoboReadingState.user_id == current_user.id,
-             ub.KoboReadingState.book_id.notin_(reading_states_in_new_entitlements)))
+             ub.KoboReadingState.book_id.notin_(reading_states_in_new_entitlements)))\
-
+        .order_by(ub.KoboReadingState.last_modified)
-    for kobo_reading_state in changed_reading_states.all():
+    cont_sync |= bool(changed_reading_states.count() > SYNC_ITEM_LIMIT)
+    for kobo_reading_state in changed_reading_states.limit(SYNC_ITEM_LIMIT).all():
         book = calibre_db.session.query(db.Books).filter(db.Books.id == kobo_reading_state.book_id).one_or_none()
         if book:
             sync_results.append({
@@ -301,18 +312,19 @@ def HandleSyncRequest():
 
     sync_shelves(sync_token, sync_results, only_kobo_shelves)
 
+    # update last created timestamp to distinguish between new and changed entitlements
+    if not cont_sync:
         sync_token.books_last_created = new_books_last_created
     sync_token.books_last_modified = new_books_last_modified
     sync_token.archive_last_modified = new_archived_last_modified
     sync_token.reading_state_last_modified = new_reading_state_last_modified
-    sync_token.books_last_id = books_last_id
 
-    return generate_sync_response(sync_token, sync_results, book_count)
+    return generate_sync_response(sync_token, sync_results, cont_sync)
 
 
 def generate_sync_response(sync_token, sync_results, set_cont=False):
     extra_headers = {}
-    if config.config_kobo_proxy:
+    if config.config_kobo_proxy and not set_cont:
         # Merge in sync results from the official Kobo store.
         try:
             store_response = make_request_to_kobo_store(sync_token)
@@ -330,9 +342,10 @@ def generate_sync_response(sync_token, sync_results, set_cont=False):
         extra_headers["x-kobo-sync"] = "continue"
     sync_token.to_headers(extra_headers)
 
-    log.debug("Kobo Sync Content: {}".format(sync_results))
+    # log.debug("Kobo Sync Content: {}".format(sync_results))
-    response = make_response(jsonify(sync_results), extra_headers)
+    # jsonify decodes the unicode string different to what kobo expects
-
+    response = make_response(json.dumps(sync_results), extra_headers)
+    response.headers["Content-Type"] = "application/json; charset=utf-8"
     return response
 
 
@@ -349,7 +362,9 @@ def HandleMetadataRequest(book_uuid):
         return redirect_or_proxy_request()
 
     metadata = get_metadata(book)
-    return jsonify([metadata])
+    response = make_response(json.dumps([metadata], ensure_ascii=False))
+    response.headers["Content-Type"] = "application/json; charset=utf-8"
+    return response
 
 
 def get_download_url_for_book(book, book_format):
@@ -377,7 +392,7 @@ def get_download_url_for_book(book, book_format):
 
 
 def create_book_entitlement(book, archived):
-    book_uuid = book.uuid
+    book_uuid = str(book.uuid)
     return {
         "Accessibility": "Full",
         "ActivePeriod": {"From": convert_to_kobo_timestamp_string(datetime.datetime.now())},
@@ -404,18 +419,15 @@ def get_description(book):
     return book.comments[0].text
 
 
-# TODO handle multiple authors
 def get_author(book):
     if not book.authors:
         return {"Contributors": None}
-    if len(book.authors) > 1:
     author_list = []
     autor_roles = []
     for author in book.authors:
-            autor_roles.append({"Name":author.name, "Role":"Author"})
+        autor_roles.append({"Name":author.name})    #.encode('unicode-escape').decode('latin-1')
         author_list.append(author.name)
     return {"ContributorRoles": autor_roles, "Contributors":author_list}
-    return {"ContributorRoles": [{"Name":book.authors[0].name, "Role":"Author"}], "Contributors": book.authors[0].name}
 
 
 def get_publisher(book):
@@ -472,9 +484,7 @@ def get_metadata(book):
         "IsSocialEnabled": True,
         "Language": "en",
         "PhoneticPronunciations": {},
-        # TODO: Fix book.pubdate to return a datetime object so that we can easily
+        "PublicationDate": convert_to_kobo_timestamp_string(book.pubdate),
-        # convert it to the format Kobo devices expect.
-        "PublicationDate": book.pubdate,
         "Publisher": {"Imprint": "", "Name": get_publisher(book),},
         "RevisionId": book_uuid,
         "Title": book.title,
@@ -483,21 +493,18 @@ def get_metadata(book):
     metadata.update(get_author(book))
 
     if get_series(book):
-        if sys.version_info < (3, 0):
-            name = get_series(book).encode("utf-8")
-        else:
         name = get_series(book)
         metadata["Series"] = {
             "Name": get_series(book),
             "Number": get_seriesindex(book),        # ToDo Check int() ?
             "NumberFloat": float(get_seriesindex(book)),
             # Get a deterministic id based on the series name.
-            "Id": uuid.uuid3(uuid.NAMESPACE_DNS, name),
+            "Id": str(uuid.uuid3(uuid.NAMESPACE_DNS, name)),
         }
 
     return metadata
 
-
+@csrf.exempt
 @kobo.route("/v1/library/tags", methods=["POST", "DELETE"])
 @requires_kobo_auth
 # Creates a Shelf with the given items, and returns the shelf's uuid.
@@ -532,6 +539,7 @@ def HandleTagCreate():
     return make_response(jsonify(str(shelf.uuid)), 201)
 
 
+@csrf.exempt
 @kobo.route("/v1/library/tags/<tag_id>", methods=["DELETE", "PUT"])
 @requires_kobo_auth
 def HandleTagUpdate(tag_id):
@@ -587,6 +595,7 @@ def add_items_to_shelf(items, shelf):
     return items_unknown_to_calibre
 
 
+@csrf.exempt
 @kobo.route("/v1/library/tags/<tag_id>/items", methods=["POST"])
 @requires_kobo_auth
 def HandleTagAddItem(tag_id):
@@ -616,6 +625,7 @@ def HandleTagAddItem(tag_id):
     return make_response('', 201)
 
 
+@csrf.exempt
 @kobo.route("/v1/library/tags/<tag_id>/items/delete", methods=["POST"])
 @requires_kobo_auth
 def HandleTagRemoveItem(tag_id):
@@ -664,11 +674,8 @@ def HandleTagRemoveItem(tag_id):
 # Note: Public shelves that aren't owned by the user aren't supported.
 def sync_shelves(sync_token, sync_results, only_kobo_shelves=False):
     new_tags_last_modified = sync_token.tags_last_modified
-
+    # transmit all archived shelfs independent of last sync (why should this matter?)
-    for shelf in ub.session.query(ub.ShelfArchive).filter(
+    for shelf in ub.session.query(ub.ShelfArchive).filter(ub.ShelfArchive.user_id == current_user.id):
-        func.datetime(ub.ShelfArchive.last_modified) > sync_token.tags_last_modified,
-        ub.ShelfArchive.user_id == current_user.id
-    ):
         new_tags_last_modified = max(shelf.last_modified, new_tags_last_modified)
         sync_results.append({
             "DeletedTag": {
@@ -678,6 +685,8 @@ def sync_shelves(sync_token, sync_results, only_kobo_shelves=False):
                 }
             }
         })
+        ub.session.delete(shelf)
+        ub.session_commit()
 
     extra_filters = []
     if only_kobo_shelves:
@@ -757,7 +766,7 @@ def create_kobo_tag(shelf):
         )
     return {"Tag": tag}
 
-
+@csrf.exempt
 @kobo.route("/v1/library/<book_uuid>/state", methods=["GET", "PUT"])
 @requires_kobo_auth
 def HandleStateRequest(book_uuid):
@@ -841,7 +850,7 @@ def get_ub_read_status(kobo_read_status):
 
 def get_or_create_reading_state(book_id):
     book_read = ub.session.query(ub.ReadBook).filter(ub.ReadBook.book_id == book_id,
-                                                          ub.ReadBook.user_id == current_user.id).one_or_none()
+                                                          ub.ReadBook.user_id == int(current_user.id)).one_or_none()
     if not book_read:
         book_read = ub.ReadBook(user_id=current_user.id, book_id=book_id)
     if not book_read.kobo_reading_state:
@@ -932,6 +941,7 @@ def TopLevelEndpoint():
     return make_response(jsonify({}))
 
 
+@csrf.exempt
 @kobo.route("/v1/library/<book_uuid>", methods=["DELETE"])
 @requires_kobo_auth
 def HandleBookDeletionRequest(book_uuid):
@@ -954,10 +964,13 @@ def HandleBookDeletionRequest(book_uuid):
 
     ub.session.merge(archived_book)
     ub.session_commit()
-    return ("", 204)
+    if archived_book.is_archived:
+        kobo_sync_status.remove_synced_book(book_id)
+    return "", 204
 
 
 # TODO: Implement the following routes
+@csrf.exempt
 @kobo.route("/v1/library/<dummy>", methods=["DELETE", "GET"])
 def HandleUnimplementedRequest(dummy=None):
     log.debug("Unimplemented Library Request received: %s", request.base_url)
@@ -965,6 +978,7 @@ def HandleUnimplementedRequest(dummy=None):
 
 
 # TODO: Implement the following routes
+@csrf.exempt
 @kobo.route("/v1/user/loyalty/<dummy>", methods=["GET", "POST"])
 @kobo.route("/v1/user/profile", methods=["GET", "POST"])
 @kobo.route("/v1/user/wishlist", methods=["GET", "POST"])
@@ -975,15 +989,41 @@ def HandleUserRequest(dummy=None):
     return redirect_or_proxy_request()
 
 
+@csrf.exempt
+@kobo.route("/v1/user/loyalty/benefits", methods=["GET"])
+def handle_benefits():
+    if config.config_kobo_proxy:
+        return redirect_or_proxy_request()
+    else:
+        return make_response(jsonify({"Benefits": {}}))
+
+
+@csrf.exempt
+@kobo.route("/v1/analytics/gettests", methods=["GET", "POST"])
+def handle_getests():
+    if config.config_kobo_proxy:
+        return redirect_or_proxy_request()
+    else:
+        testkey = request.headers.get("X-Kobo-userkey","")
+        return make_response(jsonify({"Result": "Success", "TestKey":testkey, "Tests": {}}))
+
+
+@csrf.exempt
 @kobo.route("/v1/products/<dummy>/prices", methods=["GET", "POST"])
 @kobo.route("/v1/products/<dummy>/recommendations", methods=["GET", "POST"])
 @kobo.route("/v1/products/<dummy>/nextread", methods=["GET", "POST"])
 @kobo.route("/v1/products/<dummy>/reviews", methods=["GET", "POST"])
+@kobo.route("/v1/products/featured/<dummy>", methods=["GET", "POST"])
+@kobo.route("/v1/products/featured/", methods=["GET", "POST"])
 @kobo.route("/v1/products/books/external/<dummy>", methods=["GET", "POST"])
 @kobo.route("/v1/products/books/series/<dummy>", methods=["GET", "POST"])
 @kobo.route("/v1/products/books/<dummy>", methods=["GET", "POST"])
+@kobo.route("/v1/products/books/<dummy>/", methods=["GET", "POST"])
 @kobo.route("/v1/products/dailydeal", methods=["GET", "POST"])
+@kobo.route("/v1/products/deals", methods=["GET", "POST"])
 @kobo.route("/v1/products", methods=["GET", "POST"])
+@kobo.route("/v1/affiliate", methods=["GET", "POST"])
+@kobo.route("/v1/deals", methods=["GET", "POST"])
 def HandleProductsRequest(dummy=None):
     log.debug("Unimplemented Products Request received: %s", request.base_url)
     return redirect_or_proxy_request()
@@ -1008,6 +1048,7 @@ def make_calibre_web_auth_response():
     )
 
 
+@csrf.exempt
 @kobo.route("/v1/auth/device", methods=["POST"])
 @requires_kobo_auth
 def HandleAuthRequest():

cps/kobo_auth.py

@@ -62,6 +62,7 @@ particular calls to non-Kobo specific endpoints such as the CalibreWeb book down
 from binascii import hexlify
 from datetime import datetime
 from os import urandom
+from functools import wraps
 
 from flask import g, Blueprint, url_for, abort, request
 from flask_login import login_user, current_user, login_required
@@ -70,11 +71,6 @@ from flask_babel import gettext as _
 from . import logger, config, calibre_db, db, helper, ub, lm
 from .render_template import render_title_template
 
-try:
-    from functools import wraps
-except ImportError:
-    pass  # We're not using Python 3
-
 
 log = logger.create()
 
@@ -122,20 +118,16 @@ kobo_auth = Blueprint("kobo_auth", __name__, url_prefix="/kobo_auth")
 @kobo_auth.route("/generate_auth_token/<int:user_id>")
 @login_required
 def generate_auth_token(user_id):
+    warning = False
     host_list = request.host.rsplit(':')
     if len(host_list) == 1:
         host = ':'.join(host_list)
     else:
         host = ':'.join(host_list[0:-1])
-    if host.startswith('127.') or host.lower() == 'localhost' or host.startswith('[::ffff:7f'):
+    if host.startswith('127.') or host.lower() == 'localhost' or host.startswith('[::ffff:7f') or host == "[::1]":
-        warning = _('PLease access calibre-web from non localhost to get valid api_endpoint for kobo device')
+        warning = _('Please access Calibre-Web from non localhost to get valid api_endpoint for kobo device')
-        return render_title_template(
+
-            "generate_kobo_auth_url.html",
+    # Generate auth token if none is existing for this user
-            title=_(u"Kobo Setup"),
-            warning = warning
-        )
-    else:
-        # Invalidate any prevously generated Kobo Auth token for this user.
     auth_token = ub.session.query(ub.RemoteAuthToken).filter(
         ub.RemoteAuthToken.user_id == user_id
     ).filter(ub.RemoteAuthToken.token_type==1).first()
@@ -160,17 +152,15 @@ def generate_auth_token(user_id):
     return render_title_template(
         "generate_kobo_auth_url.html",
         title=_(u"Kobo Setup"),
-            kobo_auth_url=url_for(
+        auth_token=auth_token.auth_token,
-                "kobo.TopLevelEndpoint", auth_token=auth_token.auth_token, _external=True
+        warning = warning
-            ),
-            warning = False
     )
 
 
-@kobo_auth.route("/deleteauthtoken/<int:user_id>")
+@kobo_auth.route("/deleteauthtoken/<int:user_id>", methods=["POST"])
 @login_required
 def delete_auth_token(user_id):
-    # Invalidate any prevously generated Kobo Auth token for this user.
+    # Invalidate any previously generated Kobo Auth token for this user
     ub.session.query(ub.RemoteAuthToken).filter(ub.RemoteAuthToken.user_id == user_id)\
         .filter(ub.RemoteAuthToken.token_type==1).delete()
 

cps/kobo_sync_status.py

@@ -0,0 +1,88 @@
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2021 OzzieIsaacs
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+from flask_login import current_user
+from . import ub
+import datetime
+from sqlalchemy.sql.expression import or_, and_, true
+from sqlalchemy import exc
+
+# Add the current book id to kobo_synced_books table for current user, if entry is already present,
+# do nothing (safety precaution)
+def add_synced_books(book_id):
+    is_present = ub.session.query(ub.KoboSyncedBooks).filter(ub.KoboSyncedBooks.book_id == book_id)\
+        .filter(ub.KoboSyncedBooks.user_id == current_user.id).count()
+    if not is_present:
+        synced_book = ub.KoboSyncedBooks()
+        synced_book.user_id = current_user.id
+        synced_book.book_id = book_id
+        ub.session.add(synced_book)
+        ub.session_commit()
+
+
+# Select all entries of current book in kobo_synced_books table, which are from current user and delete them
+def remove_synced_book(book_id, all=False, session=None):
+    if not all:
+        user = ub.KoboSyncedBooks.user_id == current_user.id
+    else:
+        user = true()
+    if not session:
+        ub.session.query(ub.KoboSyncedBooks).filter(ub.KoboSyncedBooks.book_id == book_id).filter(user).delete()
+        ub.session_commit()
+    else:
+        session.query(ub.KoboSyncedBooks).filter(ub.KoboSyncedBooks.book_id == book_id).filter(user).delete()
+        ub.session_commit(sess=session)
+
+
+
+def change_archived_books(book_id, state=None, message=None):
+    archived_book = ub.session.query(ub.ArchivedBook).filter(and_(ub.ArchivedBook.user_id == int(current_user.id),
+                                                                  ub.ArchivedBook.book_id == book_id)).first()
+    if not archived_book:
+        archived_book = ub.ArchivedBook(user_id=current_user.id, book_id=book_id)
+
+    archived_book.is_archived = state if state else not archived_book.is_archived
+    archived_book.last_modified = datetime.datetime.utcnow()
+
+    ub.session.merge(archived_book)
+    ub.session_commit(message)
+    return archived_book.is_archived
+
+
+# select all books which are synced by the current user and do not belong to a synced shelf and set them to archive
+# select all shelves from current user which are synced and do not belong to the "only sync" shelves
+def update_on_sync_shelfs(user_id):
+    books_to_archive = (ub.session.query(ub.KoboSyncedBooks)
+                        .join(ub.BookShelf, ub.KoboSyncedBooks.book_id == ub.BookShelf.book_id, isouter=True)
+                        .join(ub.Shelf, ub.Shelf.user_id == user_id, isouter=True)
+                        .filter(or_(ub.Shelf.kobo_sync == 0, ub.Shelf.kobo_sync == None))
+                        .filter(ub.KoboSyncedBooks.user_id == user_id).all())
+    for b in books_to_archive:
+        change_archived_books(b.book_id, True)
+        ub.session.query(ub.KoboSyncedBooks) \
+            .filter(ub.KoboSyncedBooks.book_id == b.book_id) \
+            .filter(ub.KoboSyncedBooks.user_id == user_id).delete()
+        ub.session_commit()
+
+    # Search all shelf which are currently not synced
+    shelves_to_archive = ub.session.query(ub.Shelf).filter(ub.Shelf.user_id == user_id).filter(
+        ub.Shelf.kobo_sync == 0).all()
+    for a in shelves_to_archive:
+        ub.session.add(ub.ShelfArchive(uuid=a.uuid, user_id=user_id))
+        ub.session_commit()

cps/logger.py

@@ -16,7 +16,6 @@
 #   You should have received a copy of the GNU General Public License
 #   along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import os
 import sys
 import inspect
@@ -43,22 +42,17 @@ logging.addLevelName(logging.CRITICAL, "CRIT")
 
 class _Logger(logging.Logger):
 
-    def debug_or_exception(self, message, *args, **kwargs):
+    def debug_or_exception(self, message, stacklevel=2, *args, **kwargs):
         if sys.version_info > (3, 7):
             if is_debug_enabled():
-                self.exception(message, stacklevel=2, *args, **kwargs)
+                self.exception(message, stacklevel=stacklevel, *args, **kwargs)
+            else:
+                self.error(message, stacklevel=stacklevel, *args, **kwargs)
         else:
-                self.error(message, stacklevel=2, *args, **kwargs)
-        elif sys.version_info > (3, 0):
             if is_debug_enabled():
                 self.exception(message, stack_info=True, *args, **kwargs)
             else:
                 self.error(message, *args, **kwargs)
-        else:
-            if is_debug_enabled():
-                self.exception(message, *args, **kwargs)
-            else:
-                self.error(message, *args, **kwargs)
 
 
     def debug_no_auth(self, message, *args, **kwargs):

cps/metadata_provider/comicvine.py

@@ -0,0 +1,65 @@
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2021 OzzieIsaacs
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# ComicVine api document: https://comicvine.gamespot.com/api/documentation
+
+import requests
+from cps.services.Metadata import Metadata
+
+
+class ComicVine(Metadata):
+    __name__ = "ComicVine"
+    __id__ = "comicvine"
+
+    def search(self, query, generic_cover=""):
+        val = list()
+        apikey = "57558043c53943d5d1e96a9ad425b0eb85532ee6"
+        if self.active:
+            headers = {
+                'User-Agent': 'Not Evil Browser'
+            }
+
+            result = requests.get("https://comicvine.gamespot.com/api/search?api_key="
+                                  + apikey + "&resources=issue&query=" + query + "&sort=name:desc&format=json", headers=headers)
+            for r in result.json().get('results'):
+                seriesTitle = r['volume'].get('name', "")
+                if r.get('store_date'):
+                    dateFomers = r.get('store_date')
+                else:
+                    dateFomers = r.get('date_added')
+                v = dict()
+                v['id'] = r['id']
+                v['title'] = seriesTitle + " #" + r.get('issue_number', "0") + " - " + ( r.get('name', "") or "")
+                v['authors'] = r.get('authors', [])
+                v['description'] = r.get('description', "")
+                v['publisher'] = ""
+                v['publishedDate'] = dateFomers
+                v['tags'] = ["Comics", seriesTitle]
+                v['rating'] = 0
+                v['series'] = seriesTitle
+                v['cover'] = r['image'].get('original_url')
+                v['source'] = {
+                    "id": self.__id__,
+                    "description": "ComicVine Books",
+                    "link": "https://comicvine.gamespot.com/"
+                }
+                v['url'] = r.get('site_detail_url', "")
+                val.append(v)
+        return val
+
+

cps/metadata_provider/google.py

@@ -0,0 +1,55 @@
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2021 OzzieIsaacs
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+# Google Books api document: https://developers.google.com/books/docs/v1/using
+
+
+import requests
+from cps.services.Metadata import Metadata
+
+class Google(Metadata):
+    __name__ = "Google"
+    __id__ = "google"
+
+    def search(self, query, generic_cover=""):
+        if self.active:
+            val = list()
+            result = requests.get("https://www.googleapis.com/books/v1/volumes?q="+query.replace(" ","+"))
+            for r in result.json().get('items'):
+                v = dict()
+                v['id'] = r['id']
+                v['title'] = r['volumeInfo'].get('title',"")
+                v['authors'] = r['volumeInfo'].get('authors', [])
+                v['description'] = r['volumeInfo'].get('description', "")
+                v['publisher'] = r['volumeInfo'].get('publisher', "")
+                v['publishedDate'] = r['volumeInfo'].get('publishedDate', "")
+                v['tags'] = r['volumeInfo'].get('categories', [])
+                v['rating'] = r['volumeInfo'].get('averageRating', 0)
+                if r['volumeInfo'].get('imageLinks'):
+                    v['cover'] = r['volumeInfo']['imageLinks']['thumbnail'].replace("http://", "https://")
+                else:
+                    v['cover'] = "/../../../static/generic_cover.jpg"
+                v['source'] = {
+                    "id": self.__id__,
+                    "description": "Google Books",
+                    "link": "https://books.google.com/"}
+                v['url'] = "https://books.google.com/books?id=" + r['id']
+                val.append(v)
+            return val
+
+

cps/metadata_provider/scholar.py

@@ -0,0 +1,60 @@
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2021 OzzieIsaacs
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+from scholarly import scholarly
+
+from cps.services.Metadata import Metadata
+
+class scholar(Metadata):
+    __name__ = "Google Scholar"
+    __id__ = "googlescholar"
+
+    def search(self, query, generic_cover=""):
+        val = list()
+        if self.active:
+            scholar_gen = scholarly.search_pubs(' '.join(query.split('+')))
+            i = 0
+            for publication in scholar_gen:
+                v = dict()
+                v['id'] = publication['url_scholarbib'].split(':')[1]
+                v['title'] = publication['bib'].get('title')
+                v['authors'] = publication['bib'].get('author', [])
+                v['description'] = publication['bib'].get('abstract', "")
+                v['publisher'] = publication['bib'].get('venue', "")
+                if publication['bib'].get('pub_year'):
+                    v['publishedDate'] = publication['bib'].get('pub_year')+"-01-01"
+                else:
+                    v['publishedDate'] = ""
+                v['tags'] = []
+                v['rating'] = 0
+                v['series'] = ""
+                v['cover'] = ""
+                v['url'] = publication.get('pub_url') or publication.get('eprint_url') or "",
+                v['source'] = {
+                    "id": self.__id__,
+                    "description": "Google Scholar",
+                    "link": "https://scholar.google.com/"
+                }
+                val.append(v)
+                i += 1
+                if (i >= 10):
+                    break
+        return val
+
+
+

cps/oauth.py

@@ -16,7 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>
 
-from __future__ import division, print_function, unicode_literals
 from flask import session
 
 try:

cps/oauth_bb.py

@@ -20,7 +20,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>
 
-from __future__ import division, print_function, unicode_literals
 import json
 from functools import wraps
 

cps/opds.py

@@ -20,8 +20,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
-import sys
 import datetime
 from functools import wraps
 
@@ -36,8 +34,6 @@ from .pagination import Pagination
 from .web import render_read_books
 from .usermanagement import load_user_from_request
 from flask_babel import gettext as _
-from babel import Locale as LC
-from babel.core import UnknownLocaleError
 
 opds = Blueprint('opds', __name__)
 
@@ -433,16 +429,17 @@ def feed_languagesindex():
     if current_user.filter_language() == u"all":
         languages = calibre_db.speaking_language()
     else:
-        try:
+        #try:
-            cur_l = LC.parse(current_user.filter_language())
+        #    cur_l = LC.parse(current_user.filter_language())
-        except UnknownLocaleError:
+        #except UnknownLocaleError:
-            cur_l = None
+        #    cur_l = None
         languages = calibre_db.session.query(db.Languages).filter(
             db.Languages.lang_code == current_user.filter_language()).all()
-        if cur_l:
+        languages[0].name = isoLanguages.get_language_name(get_locale(), languages[0].lang_code)
-            languages[0].name = cur_l.get_language_name(get_locale())
+        #if cur_l:
-        else:
+        #    languages[0].name = cur_l.get_language_name(get_locale())
-            languages[0].name = _(isoLanguages.get(part3=languages[0].lang_code).name)
+        #else:
+        #    languages[0].name = _(isoLanguages.get(part3=languages[0].lang_code).name)
     pagination = Pagination((int(off) / (int(config.config_books_per_page)) + 1), config.config_books_per_page,
                             len(languages))
     return render_xml_template('feed.xml', listelements=languages, folder='opds.feed_languages', pagination=pagination)
@@ -536,7 +533,6 @@ def feed_search(term):
 
 
 def check_auth(username, password):
-    if sys.version_info.major == 3:
     try:
         username = username.encode('windows-1252')
     except UnicodeEncodeError:

cps/pagination.py

@@ -20,7 +20,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 from math import ceil
 
 

cps/redirect.py

@@ -27,11 +27,8 @@
 
 # http://flask.pocoo.org/snippets/62/
 
-from __future__ import division, print_function, unicode_literals
+from urllib.parse import urlparse, urljoin
-try:
+
-    from urllib.parse import urlparse, urljoin
-except ImportError:
-    from urlparse import urlparse, urljoin
 
 from flask import request, url_for, redirect
 

cps/reverseproxy.py

@@ -36,8 +36,6 @@
 #
 # Inspired by http://flask.pocoo.org/snippets/35/
 
-from __future__ import division, print_function, unicode_literals
-
 
 class ReverseProxied(object):
     """Wrap the application in this middleware and configure the

cps/search_metadata.py

@@ -0,0 +1,117 @@
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2021 OzzieIsaacs
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import json
+import importlib
+import sys
+import inspect
+import datetime
+import concurrent.futures
+
+from flask import Blueprint, request, Response, url_for
+from flask_login import current_user
+from flask_login import login_required
+from sqlalchemy.orm.attributes import flag_modified
+from sqlalchemy.exc import OperationalError, InvalidRequestError
+
+from . import constants, logger, ub
+from cps.services.Metadata import Metadata
+
+
+meta = Blueprint('metadata', __name__)
+
+log = logger.create()
+
+new_list = list()
+meta_dir = os.path.join(constants.BASE_DIR, "cps", "metadata_provider")
+modules = os.listdir(os.path.join(constants.BASE_DIR, "cps", "metadata_provider"))
+for f in modules:
+    if os.path.isfile(os.path.join(meta_dir, f)) and not f.endswith('__init__.py'):
+        a = os.path.basename(f)[:-3]
+        try:
+            importlib.import_module("cps.metadata_provider." + a)
+            new_list.append(a)
+        except ImportError:
+            log.error("Import error for metadata source: {}".format(a))
+            pass
+
+def list_classes(provider_list):
+    classes = list()
+    for element in provider_list:
+        for name, obj in inspect.getmembers(sys.modules["cps.metadata_provider." + element]):
+            if inspect.isclass(obj) and name != "Metadata" and issubclass(obj, Metadata):
+                classes.append(obj())
+    return classes
+
+cl = list_classes(new_list)
+
+@meta.route("/metadata/provider")
+@login_required
+def metadata_provider():
+    active = current_user.view_settings.get('metadata', {})
+    provider = list()
+    for c in cl:
+        ac = active.get(c.__id__, True)
+        provider.append({"name": c.__name__, "active": ac, "initial": ac, "id": c.__id__})
+    return Response(json.dumps(provider), mimetype='application/json')
+
+@meta.route("/metadata/provider", methods=['POST'])
+@meta.route("/metadata/provider/<prov_name>", methods=['POST'])
+@login_required
+def metadata_change_active_provider(prov_name):
+    new_state = request.get_json()
+    active = current_user.view_settings.get('metadata', {})
+    active[new_state['id']] = new_state['value']
+    current_user.view_settings['metadata'] = active
+    try:
+        try:
+            flag_modified(current_user, "view_settings")
+        except AttributeError:
+            pass
+        ub.session.commit()
+    except (InvalidRequestError, OperationalError):
+        log.error("Invalid request received: {}".format(request))
+        return "Invalid request", 400
+    if "initial" in new_state and prov_name:
+        for c in cl:
+            if c.__id__ == prov_name:
+                data = c.search(new_state.get('query', ""))
+                break
+        return Response(json.dumps(data), mimetype='application/json')
+    return ""
+
+@meta.route("/metadata/search", methods=['POST'])
+@login_required
+def metadata_search():
+    query = request.form.to_dict().get('query')
+    data = list()
+    active = current_user.view_settings.get('metadata', {})
+    if query:
+        generic_cover = ""
+        with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
+            meta = {executor.submit(c.search, query, generic_cover): c for c in cl if active.get(c.__id__, True)}
+            for future in concurrent.futures.as_completed(meta):
+                data.extend(future.result())
+    return Response(json.dumps(data), mimetype='application/json')
+
+
+
+
+
+

cps/server.py

@@ -16,7 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import sys
 import os
 import errno

cps/services/Metadata.py

@@ -0,0 +1,27 @@
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2021 OzzieIsaacs
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+
+class Metadata():
+    __name__ = "Generic"
+
+    def __init__(self):
+        self.active = True
+
+    def set_status(self, state):
+        self.active = state

cps/services/SyncToken.py

@@ -21,11 +21,8 @@ import sys
 from base64 import b64decode, b64encode
 from jsonschema import validate, exceptions, __version__
 from datetime import datetime
-try:
+
-    # pylint: disable=unused-import
+from urllib.parse import unquote
-    from urllib import unquote
-except ImportError:
-    from urllib.parse import unquote
 
 from flask import json
 from .. import logger
@@ -35,9 +32,6 @@ log = logger.create()
 
 
 def b64encode_json(json_data):
-    if sys.version_info < (3, 0):
-        return b64encode(json.dumps(json_data))
-    else:
     return b64encode(json.dumps(json_data).encode())
 
 
@@ -85,8 +79,8 @@ class SyncToken:
             "books_last_created": {"type": "string"},
             "archive_last_modified": {"type": "string"},
             "reading_state_last_modified": {"type": "string"},
-            "tags_last_modified": {"type": "string"},
+            "tags_last_modified": {"type": "string"}
-            "books_last_id": {"type": "integer", "optional": True}
+            # "books_last_id": {"type": "integer", "optional": True}
         },
     }
 
@@ -97,8 +91,8 @@ class SyncToken:
         books_last_modified=datetime.min,
         archive_last_modified=datetime.min,
         reading_state_last_modified=datetime.min,
-        tags_last_modified=datetime.min,
+        tags_last_modified=datetime.min
-        books_last_id=-1
+        # books_last_id=-1
     ):  # nosec
         self.raw_kobo_store_token = raw_kobo_store_token
         self.books_last_created = books_last_created
@@ -106,7 +100,7 @@ class SyncToken:
         self.archive_last_modified = archive_last_modified
         self.reading_state_last_modified = reading_state_last_modified
         self.tags_last_modified = tags_last_modified
-        self.books_last_id = books_last_id
+        # self.books_last_id = books_last_id
 
     @staticmethod
     def from_headers(headers):
@@ -141,12 +135,12 @@ class SyncToken:
             archive_last_modified = get_datetime_from_json(data_json, "archive_last_modified")
             reading_state_last_modified = get_datetime_from_json(data_json, "reading_state_last_modified")
             tags_last_modified = get_datetime_from_json(data_json, "tags_last_modified")
-            books_last_id = data_json["books_last_id"]
+            # books_last_id = data_json["books_last_id"]
         except TypeError:
             log.error("SyncToken timestamps don't parse to a datetime.")
             return SyncToken(raw_kobo_store_token=raw_kobo_store_token)
-        except KeyError:
+        #except KeyError:
-            books_last_id = -1
+        #    books_last_id = -1
 
         return SyncToken(
             raw_kobo_store_token=raw_kobo_store_token,
@@ -155,7 +149,7 @@ class SyncToken:
             archive_last_modified=archive_last_modified,
             reading_state_last_modified=reading_state_last_modified,
             tags_last_modified=tags_last_modified,
-            books_last_id=books_last_id
+            #books_last_id=books_last_id
         )
 
     def set_kobo_store_header(self, store_headers):
@@ -179,16 +173,15 @@ class SyncToken:
                 "archive_last_modified": to_epoch_timestamp(self.archive_last_modified),
                 "reading_state_last_modified": to_epoch_timestamp(self.reading_state_last_modified),
                 "tags_last_modified": to_epoch_timestamp(self.tags_last_modified),
-                "books_last_id":self.books_last_id
+                #"books_last_id":self.books_last_id
             },
         }
         return b64encode_json(token)
 
     def __str__(self):
-        return "{},{},{},{},{},{},{}".format(self.raw_kobo_store_token,
+        return "{},{},{},{},{},{}".format(self.books_last_created,
-                                       self.books_last_created,
                                        self.books_last_modified,
                                        self.archive_last_modified,
                                        self.reading_state_last_modified,
-                                       self.tags_last_modified,
+                                       self.tags_last_modified, self.raw_kobo_store_token)
-                                       self.books_last_id)
+                                       #self.books_last_id)

cps/services/__init__.py

@@ -16,8 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
-
 from .. import logger
 
 

cps/services/gmail.py

@@ -1,4 +1,21 @@
-from __future__ import print_function
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2021 OzzieIsaacs
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
 import os.path
 from google_auth_oauthlib.flow import InstalledAppFlow
 from google.auth.transport.requests import Request

cps/services/goodreads_support.py

@@ -16,7 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import time
 from functools import reduce
 

cps/services/simpleldap.py

@@ -16,7 +16,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import base64
 
 from flask_simpleldap import LDAP, LDAPException

cps/services/worker.py

@@ -1,5 +1,21 @@
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2020 pwr
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
 import threading
 import abc
 import uuid
@@ -69,7 +85,7 @@ class WorkerThread(threading.Thread):
     def add(cls, user, task):
         ins = cls.getInstance()
         ins.num += 1
-        log.debug("Add Task for user: {}: {}".format(user, task))
+        log.debug("Add Task for user: {} - {}".format(user, task))
         ins.queue.put(QueuedTask(
             num=ins.num,
             user=user,
@@ -205,10 +221,13 @@ class CalibreTask:
         # By default, we're good to clean a task if it's "Done"
         return self.stat in (STAT_FINISH_SUCCESS, STAT_FAIL)
 
-    @progress.setter
+    '''@progress.setter
     def progress(self, x):        
-        # todo: throw error if outside of [0,1]
+        if x > 1: 
-        self._progress = x
+            x = 1
+        if x < 0: 
+            x = 0
+        self._progress = x'''
 
     def _handleError(self, error_message):
         self.stat = STAT_FAIL

cps/shelf.py

@@ -20,8 +20,6 @@
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
-from __future__ import division, print_function, unicode_literals
-
 import sys
 from datetime import datetime
 
@@ -58,7 +56,7 @@ def check_shelf_view_permissions(cur_shelf):
     return True
 
 
-@shelf.route("/shelf/add/<int:shelf_id>/<int:book_id>")
+@shelf.route("/shelf/add/<int:shelf_id>/<int:book_id>", methods=["POST"])
 @login_required
 def add_to_shelf(shelf_id, book_id):
     xhr = request.headers.get('X-Requested-With') == 'XMLHttpRequest'
@@ -114,7 +112,7 @@ def add_to_shelf(shelf_id, book_id):
     return "", 204
 
 
-@shelf.route("/shelf/massadd/<int:shelf_id>")
+@shelf.route("/shelf/massadd/<int:shelf_id>", methods=["POST"])
 @login_required
 def search_to_shelf(shelf_id):
     shelf = ub.session.query(ub.Shelf).filter(ub.Shelf.id == shelf_id).first()
@@ -124,8 +122,8 @@ def search_to_shelf(shelf_id):
         return redirect(url_for('web.index'))
 
     if not check_shelf_edit_permissions(shelf):
-        log.warning("You are not allowed to add a book to the the shelf: {}".format(shelf.name))
+        log.warning("You are not allowed to add a book to the shelf".format(shelf.name))
-        flash(_(u"You are not allowed to add a book to the the shelf: %(name)s", name=shelf.name), category="error")
+        flash(_(u"You are not allowed to add a book to the shelf"), category="error")
         return redirect(url_for('web.index'))
 
     if current_user.id in ub.searched_ids and ub.searched_ids[current_user.id]:
@@ -166,7 +164,7 @@ def search_to_shelf(shelf_id):
     return redirect(url_for('web.index'))
 
 
-@shelf.route("/shelf/remove/<int:shelf_id>/<int:book_id>")
+@shelf.route("/shelf/remove/<int:shelf_id>/<int:book_id>", methods=["POST"])
 @login_required
 def remove_from_shelf(shelf_id, book_id):
     xhr = request.headers.get('X-Requested-With') == 'XMLHttpRequest'
@@ -217,10 +215,10 @@ def remove_from_shelf(shelf_id, book_id):
     else:
         if not xhr:
             log.warning("You are not allowed to remove a book from shelf: {}".format(shelf.name))
-            flash(_(u"Sorry you are not allowed to remove a book from this shelf: %(sname)s", sname=shelf.name),
+            flash(_(u"Sorry you are not allowed to remove a book from this shelf"),
                   category="error")
             return redirect(url_for('web.index'))
-        return "Sorry you are not allowed to remove a book from this shelf: %s" % shelf.name, 403
+        return "Sorry you are not allowed to remove a book from this shelf", 403
 
 
 @shelf.route("/shelf/create", methods=["GET", "POST"])
@@ -230,6 +228,7 @@ def create_shelf():
     return create_edit_shelf(shelf, page_title=_(u"Create a Shelf"), page="shelfcreate")
 
 
+
 @shelf.route("/shelf/edit/<int:shelf_id>", methods=["GET", "POST"])
 @login_required
 def edit_shelf(shelf_id):
@@ -246,12 +245,20 @@ def create_edit_shelf(shelf, page_title, page, shelf_id=False):
     # calibre_db.session.query(ub.Shelf).filter(ub.Shelf.user_id == current_user.id).filter(ub.Shelf.kobo_sync).count()
     if request.method == "POST":
         to_save = request.form.to_dict()
-        shelf.is_public = 1 if to_save.get("is_public") else 0
+        if not current_user.role_edit_shelfs() and to_save.get("is_public") == "on":
+            flash(_(u"Sorry you are not allowed to create a public shelf"), category="error")
+            return redirect(url_for('web.index'))
+        is_public = 1 if to_save.get("is_public") == "on" else 0
         if config.config_kobo_sync:
             shelf.kobo_sync = True if to_save.get("kobo_sync") else False
+            if shelf.kobo_sync:
+                ub.session.query(ub.ShelfArchive).filter(ub.ShelfArchive.user_id == current_user.id).filter(
+                    ub.ShelfArchive.uuid == shelf.uuid).delete()
+                ub.session_commit()
         shelf_title = to_save.get("title", "")
-        if check_shelf_is_unique(shelf, shelf_title, shelf_id):
+        if check_shelf_is_unique(shelf, shelf_title, is_public, shelf_id):
             shelf.name = shelf_title
+            shelf.is_public = is_public
             if not shelf_id:
                 shelf.user_id = int(current_user.id)
                 ub.session.add(shelf)
@@ -282,12 +289,12 @@ def create_edit_shelf(shelf, page_title, page, shelf_id=False):
                                  sync_only_selected_shelves=sync_only_selected_shelves)
 
 
-def check_shelf_is_unique(shelf, title, shelf_id=False):
+def check_shelf_is_unique(shelf, title, is_public, shelf_id=False):
     if shelf_id:
         ident = ub.Shelf.id != shelf_id
     else:
         ident = true()
-    if shelf.is_public == 1:
+    if is_public == 1:
         is_shelf_name_unique = ub.session.query(ub.Shelf) \
                                    .filter((ub.Shelf.name == title) & (ub.Shelf.is_public == 1)) \
                                    .filter(ident) \
@@ -321,12 +328,13 @@ def delete_shelf_helper(cur_shelf):
     ub.session_commit("successfully deleted Shelf {}".format(cur_shelf.name))
 
 
-@shelf.route("/shelf/delete/<int:shelf_id>")
+@shelf.route("/shelf/delete/<int:shelf_id>", methods=["POST"])
 @login_required
 def delete_shelf(shelf_id):
     cur_shelf = ub.session.query(ub.Shelf).filter(ub.Shelf.id == shelf_id).first()
     try:
         delete_shelf_helper(cur_shelf)
+        flash(_("Shelf successfully deleted"), category="success")
     except InvalidRequestError:
         ub.session.rollback()
         log.error("Settings DB is not Writeable")

cps/static/css/libs/Djvu_html5.css

@@ -35,7 +35,7 @@ body {
 	float: left;
 	width: 40px;
 	height: 40px;
-	background-image: url("img/toolbar-buttons.png");
+	background-image: url("../../js/libs/djvu_html5/img/toolbar-buttons.png");
 	background-repeat: no-repeat;
 	background-size: 500% 300%;
 }
@@ -185,10 +185,10 @@ _:-ms-lang(x), .textLayer {
 	width: 128px;
 	height: 128px;
 	margin: -72px 0 0 -64px;
-	background-image: url("img/status.png");
+	background-image: url("../../js/libs/djvu_html5/img/status.png");
 	background-repeat: no-repeat;
 }
 
 .blankImage {
-	background-image: url("img/blank.jpg");
+	background-image: url("../../js/libs/djvu_html5/img/blank.jpg");
 }

cps/static/css/libs/bootstrap-wysihtml5-0.0.3.css

@@ -0,0 +1,103 @@
+ul.wysihtml5-toolbar {
+	margin: 0;
+	padding: 0;
+	display: block;
+}
+
+ul.wysihtml5-toolbar::after {
+	clear: both;
+	display: table;
+	content: "";
+}
+
+ul.wysihtml5-toolbar > li {
+	float: left;
+	display: list-item;
+	list-style: none;
+	margin: 0 5px 10px 0;
+}
+
+ul.wysihtml5-toolbar a[data-wysihtml5-command=bold] {
+	font-weight: bold;
+}
+
+ul.wysihtml5-toolbar a[data-wysihtml5-command=italic] {
+	font-style: italic;
+}
+
+ul.wysihtml5-toolbar a[data-wysihtml5-command=underline] {
+	text-decoration: underline;
+}
+
+ul.wysihtml5-toolbar a.btn.wysihtml5-command-active {
+	background-image: none;
+	-webkit-box-shadow: inset 0 2px 4px rgba(0, 0, 0, 0.15),0 1px 2px rgba(0, 0, 0, 0.05);
+	-moz-box-shadow: inset 0 2px 4px rgba(0, 0, 0, 0.15),0 1px 2px rgba(0, 0, 0, 0.05);
+	box-shadow: inset 0 2px 4px rgba(0, 0, 0, 0.15),0 1px 2px rgba(0, 0, 0, 0.05);
+	background-color: #E6E6E6;
+	background-color: #D9D9D9;
+	outline: 0;
+}
+
+ul.wysihtml5-commands-disabled .dropdown-menu {
+	display: none !important;
+}
+
+ul.wysihtml5-toolbar div.wysihtml5-colors {
+  display:block;
+  width: 50px;
+  height: 20px;
+  margin-top: 2px;
+  margin-left: 5px;
+  position: absolute;
+  pointer-events: none;
+}
+
+ul.wysihtml5-toolbar a.wysihtml5-colors-title {
+  padding-left: 70px;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="black"] {
+  background: black !important;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="silver"] {
+  background: silver !important;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="gray"] {
+  background: gray !important;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="maroon"] {
+  background: maroon !important;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="red"] {
+  background: red !important;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="purple"] {
+  background: purple !important;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="green"] {
+  background: green !important;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="olive"] {
+  background: olive !important;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="navy"] {
+  background: navy !important;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="blue"] {
+  background: blue !important;
+}
+
+ul.wysihtml5-toolbar div[data-wysihtml5-command-value="orange"] {
+  background: orange !important;
+}
+

cps/static/css/libs/typeahead.css

@@ -145,7 +145,7 @@ fieldset[disabled] .twitter-typeahead .tt-input {
   cursor: not-allowed;
   background-color: #eeeeee !important;
 }
-.tt-dropdown-menu {
+.tt-menu {
   position: absolute;
   top: 100%;
   left: 0;
@@ -166,7 +166,7 @@ fieldset[disabled] .twitter-typeahead .tt-input {
   *border-right-width: 2px;
   *border-bottom-width: 2px;
 }
-.tt-dropdown-menu .tt-suggestion {
+.tt-menu .tt-suggestion {
   display: block;
   padding: 3px 20px;
   clear: both;
@@ -175,15 +175,15 @@ fieldset[disabled] .twitter-typeahead .tt-input {
   color: #333333;
   white-space: nowrap;
 }
-.tt-dropdown-menu .tt-suggestion.tt-cursor {
+.tt-menu .tt-suggestion.tt-cursor {
   text-decoration: none;
   outline: 0;
   background-color: #f5f5f5;
   color: #262626;
 }
-.tt-dropdown-menu .tt-suggestion.tt-cursor a {
+.tt-menu .tt-suggestion.tt-cursor a {
   color: #262626;
 }
-.tt-dropdown-menu .tt-suggestion p {
+.tt-menu .tt-suggestion p {
   margin: 0;
 }

cps/static/css/style.css

@@ -123,6 +123,10 @@ table .bg-dark-danger a { color: #fff; }
   flex-wrap: wrap;
 }
 
+.row-fluid.text-center {
+    margin-top: -20px;
+}
+
 .container-fluid img {
   display: block;
   max-width: 100%;
@@ -151,11 +155,6 @@ table .bg-dark-danger a { color: #fff; }
   height: 100%;
 }
 
-.container-fluid .book .cover span.img {
-  bottom: 0;
-  height: 100%;
-  position: absolute;
-}
 .author-bio img { margin: 0 1em 1em 0; }
 
 .container-fluid .single .cover img {
@@ -166,12 +165,21 @@ table .bg-dark-danger a { color: #fff; }
   box-shadow: 0 5px 8px -6px #777;
 }
 
-.container-fluid .book .cover span img {
+.datepicker.form-control {
-  position: relative;
+    position: static;
-  top: 0;
+}
-  left: 0;
+
+.container-fluid .book .cover span .img {
+  bottom: 0;
   height: 100%;
+  position: absolute;
+}
+
+.container-fluid .book .cover span img {
   border: 1px solid #fff;
+  position: relative;
+  height: 100%;
+
   box-sizing: border-box;
   -webkit-box-shadow: 0 5px 8px -6px #777;
   -moz-box-shadow: 0 5px 8px -6px #777;
@@ -243,8 +251,10 @@ span.glyphicon.glyphicon-tags {
 }
 
 .cover .read {
-  left: auto;
+  position: relative;
-  right: 2px;
+  top: -20px;
+  /*left: auto;
+  right: 2px;*/
   width: 17px;
   height: 17px;
   display: inline-block;
@@ -322,7 +332,7 @@ table .bg-dark-danger:hover { background-color: #c9302c; }
 table .bg-primary:hover { background-color: #1c5484; }
 .block-label { display: block; }
 
-.fake-input {
+.form-control.fake-input {
   position: absolute;
   pointer-events: none;
   top: 0;
@@ -419,7 +429,7 @@ div.log {
 }
 
 #detailcover { cursor:zoom-in; }
-#detailcover:-webkit-full-screen { cursor:zoom-out; }
+#detailcover:-webkit-full-screen { cursor:zoom-out; border: 0; }
-#detailcover:-moz-full-screen { cursor:zoom-out; }
+#detailcover:-moz-full-screen { cursor:zoom-out; border: 0; }
-#detailcover:-ms-fullscreen { cursor:zoom-out; }
+#detailcover:-ms-fullscreen { cursor:zoom-out; border: 0; }
-#detailcover:fullscreen { cursor:zoom-out; }
+#detailcover:fullscreen { cursor:zoom-out; border: 0; }

cps/static/css/text.css

@@ -0,0 +1,44 @@
+body {
+    background: white;
+}
+
+#readmain {
+    position: absolute;
+    width: 100%;
+    height: 100%;
+}
+
+#area {
+    width: 80%;
+    height: 80%;
+    margin: 5% auto;
+    max-width: 1250px;
+}
+
+#area iframe {
+    border: none;
+}
+
+xmp, pre, plaintext {
+    display: block;
+    font-family: -moz-fixed;
+    white-space: pre;
+    margin: 1em 0;
+}
+
+#area{
+    overflow:hidden;
+}
+
+pre {
+    white-space: pre-wrap;
+    word-wrap: break-word;
+    font-family: -moz-fixed;
+    column-count:2;
+    -webkit-columns:2;
+    -moz-columns:2;
+    column-gap:20px;
+    -moz-column-gap:20px;
+    -webkit-column-gap:20px;
+    position:relative;
+}

cps/static/js/caliBlur.js

@@ -150,11 +150,16 @@ if ($("body.book").length > 0) {
         var splitText = $(this).text().split(':');
         var label = splitText.shift().trim();
         var value = splitText.join(':').trim();
+        var class_value = ""
         // Preserve Links
         if ($(this).find('a').length) {
             value = $(this).find('a').first().removeClass();
         }
-        $(this).html('<span>' + label + '</span><span></span>').find('span').last().append(value);
+        // Preserve glyphicons
+        if ($(this).find('span').length) {
+            class_value = $(this).find('span').first().attr('class');
+        }
+        $(this).html('<span>' + label + '</span><span class="' + class_value + '"></span>').find('span').last().append(value);
     });
 
     $(".book-meta h2:first").clone()

cps/static/js/details.js

@@ -22,15 +22,17 @@ $(function() {
 });
 
 $("#have_read_cb").on("change", function() {
-    $.post({
+    $.ajax({
         url: this.closest("form").action,
+        method:"post",
+        data: $(this).closest("form").serialize(),
         error: function(response) {
             var data = [{type:"danger", message:response.responseText}]
             $("#flash_success").remove();
             $("#flash_danger").remove();
             if (!jQuery.isEmptyObject(data)) {
                 data.forEach(function (item) {
-                    $(".navbar").after('<div class="row-fluid text-center" style="margin-top: -20px;">' +
+                    $(".navbar").after('<div class="row-fluid text-center" >' +
                         '<div id="flash_' + item.type + '" class="alert alert-' + item.type + '">' + item.message + '</div>' +
                         '</div>');
                 });
@@ -59,8 +61,11 @@ $("#archived_cb").on("change", function() {
 
     $("#shelf-actions").on("click", "[data-shelf-action]", function (e) {
         e.preventDefault();
-
+        $.ajax({
-        $.get(this.href)
+                url: this.href,
+                method:"post",
+                data: {csrf_token:$("input[name='csrf_token']").val()},
+            })
             .done(function() {
                 var $this = $(this);
                 switch ($this.data("shelf-action")) {

cps/static/js/edit_books.js

@@ -6,6 +6,7 @@
 if ($("#description").length) {
     tinymce.init({
         selector: "#description",
+        plugins: 'code',
         branding: false,
         menubar: "edit view format",
         language: language
@@ -15,6 +16,7 @@ if ($("#description").length) {
 if ($(".tiny_editor").length) {
     tinymce.init({
         selector: ".tiny_editor",
+        plugins: 'code',
         branding: false,
         menubar: "edit view format",
         language: language
@@ -47,90 +49,20 @@ $(".datepicker_delete").click(function() {
 Takes a prefix, query typeahead callback, Bloodhound typeahead adapter
  and returns the completions it gets from the bloodhound engine prefixed.
  */
-function prefixedSource(prefix, query, cb, bhAdapter) {
+function prefixedSource(prefix, query, cb, source) {
-    bhAdapter(query, function(retArray) {
+    function async(retArray) {
+        retArray = retArray || [];
         var matches = [];
         for (var i = 0; i < retArray.length; i++) {
             var obj = {name : prefix + retArray[i].name};
             matches.push(obj);
         }
         cb(matches);
-    });
+    }
+    source.search(query, cb, async);
 }
 
-var authors = new Bloodhound({
-    name: "authors",
-    datumTokenizer: function datumTokenizer(datum) {
-        return [datum.name];
-    },
-    queryTokenizer: Bloodhound.tokenizers.whitespace,
-    remote: {
-        url: getPath() + "/get_authors_json?q=%QUERY"
-    }
-});
-
-var series = new Bloodhound({
-    name: "series",
-    datumTokenizer: function datumTokenizer(datum) {
-        return [datum.name];
-    },
-    queryTokenizer: function queryTokenizer(query) {
-        return [query];
-    },
-    remote: {
-        url: getPath() + "/get_series_json?q=",
-        replace: function replace(url, query) {
-            return url + encodeURIComponent(query);
-        }
-    }
-});
-
-
-var tags = new Bloodhound({
-    name: "tags",
-    datumTokenizer: function datumTokenizer(datum) {
-        return [datum.name];
-    },
-    queryTokenizer: function queryTokenizer(query) {
-        var tokens = query.split(",");
-        tokens = [tokens[tokens.length - 1].trim()];
-        return tokens;
-    },
-    remote: {
-        url: getPath() + "/get_tags_json?q=%QUERY"
-    }
-});
-
-var languages = new Bloodhound({
-    name: "languages",
-    datumTokenizer: function datumTokenizer(datum) {
-        return [datum.name];
-    },
-    queryTokenizer: function queryTokenizer(query) {
-        return [query];
-    },
-    remote: {
-        url: getPath() + "/get_languages_json?q=",
-        replace: function replace(url, query) {
-            return url + encodeURIComponent(query);
-        }
-    }
-});
-
-var publishers = new Bloodhound({
-    name: "publisher",
-    datumTokenizer: function datumTokenizer(datum) {
-        return [datum.name];
-    },
-    queryTokenizer: Bloodhound.tokenizers.whitespace,
-    remote: {
-        url: getPath() + "/get_publishers_json?q=%QUERY"
-    }
-});
-
 function sourceSplit(query, cb, split, source) {
-    var bhAdapter = source.ttAdapter();
-
     var tokens = query.split(split);
     var currentSource = tokens[tokens.length - 1].trim();
 
@@ -145,84 +77,148 @@ function sourceSplit(query, cb, split, source) {
     for (var i = 0; i < tokens.length; i++) {
         prefix += tokens[i].trim() + newSplit;
     }
-    prefixedSource(prefix, currentSource, cb, bhAdapter);
+    prefixedSource(prefix, currentSource, cb, source);
 }
 
-var promiseAuthors = authors.initialize();
+var authors = new Bloodhound({
-promiseAuthors.done(function() {
+    name: "authors",
-    $("#bookAuthor").typeahead(
+    identify: function(obj) { return obj.name; },
+    datumTokenizer: function datumTokenizer(datum) {
+        return [datum.name];
+    },
+    queryTokenizer: Bloodhound.tokenizers.whitespace,
+    remote: {
+        url: getPath() + "/get_authors_json?q=%QUERY",
+        wildcard: '%QUERY',
+    },
+});
+
+$(".form-group #bookAuthor").typeahead(
     {
-            highlight: true, minLength: 1,
+        highlight: true,
+        minLength: 1,
         hint: true
     }, {
         name: "authors",
-            displayKey: "name",
+        display: 'name',
-            source: function source(query, cb) {
+        source: function source(query, cb, asyncResults) {
-                return sourceSplit(query, cb, "&", authors); //sourceSplit //("&")
+            return sourceSplit(query, cb, "&", authors);
         }
     }
-    );
+);
-});
 
-var promiseSeries = series.initialize();
+
-promiseSeries.done(function() {
+var series = new Bloodhound({
-    $("#series").typeahead(
+    name: "series",
+    datumTokenizer: function datumTokenizer(datum) {
+        return [datum.name];
+    },
+    // queryTokenizer: Bloodhound.tokenizers.whitespace,
+    queryTokenizer: function queryTokenizer(query) {
+        return [query];
+    },
+    remote: {
+        url: getPath() + "/get_series_json?q=%QUERY",
+        wildcard: '%QUERY',
+        /*replace: function replace(url, query) {
+            return url + encodeURIComponent(query);
+        }*/
+    }
+});
+$(".form-group #series").typeahead(
     {
-            highlight: true, minLength: 0,
+        highlight: true,
+        minLength: 0,
         hint: true
     }, {
         name: "series",
         displayKey: "name",
-            source: series.ttAdapter()
+        source: series
+    }
+);
+
+var tags = new Bloodhound({
+    name: "tags",
+    datumTokenizer: function datumTokenizer(datum) {
+        return [datum.name];
+    },
+    queryTokenizer: function queryTokenizer(query) {
+        var tokens = query.split(",");
+        tokens = [tokens[tokens.length - 1].trim()];
+        return tokens;
+    },
+    remote: {
+        url: getPath() + "/get_tags_json?q=%QUERY",
+        wildcard: '%QUERY'
     }
-    );
 });
 
-var promiseTags = tags.initialize();
+$(".form-group #tags").typeahead(
-promiseTags.done(function() {
-    $("#tags").typeahead(
     {
-            highlight: true, minLength: 0,
+        highlight: true,
+        minLength: 0,
         hint: true
     }, {
         name: "tags",
-            displayKey: "name",
+        display: "name",
-            source: function source(query, cb) {
+        source: function source(query, cb, asyncResults) {
             return sourceSplit(query, cb, ",", tags);
         }
     }
-    );
+);
+
+var languages = new Bloodhound({
+    name: "languages",
+    datumTokenizer: function datumTokenizer(datum) {
+        return [datum.name];
+    },
+    queryTokenizer: function queryTokenizer(query) {
+        return [query];
+    },
+    remote: {
+        url: getPath() + "/get_languages_json?q=%QUERY",
+        wildcard: '%QUERY'
+        /*replace: function replace(url, query) {
+            return url + encodeURIComponent(query);
+        }*/
+    }
 });
 
-var promiseLanguages = languages.initialize();
+$(".form-group #languages").typeahead(
-promiseLanguages.done(function() {
-    $("#languages").typeahead(
     {
         highlight: true, minLength: 0,
         hint: true
     }, {
         name: "languages",
-            displayKey: "name",
+        display: "name",
-            source: function source(query, cb) {
+        source: function source(query, cb, asyncResults) {
-                return sourceSplit(query, cb, ",", languages); //(",")
+            return sourceSplit(query, cb, ",", languages);
         }
     }
-    );
+);
+
+var publishers = new Bloodhound({
+    name: "publisher",
+    datumTokenizer: function datumTokenizer(datum) {
+        return [datum.name];
+    },
+    queryTokenizer: Bloodhound.tokenizers.whitespace,
+    remote: {
+        url: getPath() + "/get_publishers_json?q=%QUERY",
+        wildcard: '%QUERY'
+    }
 });
 
-var promisePublishers = publishers.initialize();
+$(".form-group #publisher").typeahead(
-promisePublishers.done(function() {
-    $("#publisher").typeahead(
     {
         highlight: true, minLength: 0,
         hint: true
     }, {
         name: "publishers",
         displayKey: "name",
-            source: publishers.ttAdapter()
+        source: publishers
     }
-    );
+);
-});
 
 $("#search").on("change input.typeahead:selected", function(event) {
     if (event.target.type === "search" && event.target.tagName === "INPUT") {
@@ -252,7 +248,7 @@ $("#btn-upload-format").on("change", function () {
     if (filename.substring(3, 11) === "fakepath") {
         filename = filename.substring(12);
     } // Remove c:\fake at beginning from localhost chrome
-    $("#upload-format").html(filename);
+    $("#upload-format").text(filename);
 });
 
 $("#btn-upload-cover").on("change", function () {
@@ -260,7 +256,7 @@ $("#btn-upload-cover").on("change", function () {
     if (filename.substring(3, 11) === "fakepath") {
         filename = filename.substring(12);
     } // Remove c:\fake at beginning from localhost chrome
-    $("#upload-cover").html(filename);
+    $("#upload-cover").text(filename);
 });
 
 $("#xchange").click(function () {
@@ -269,3 +265,4 @@ $("#xchange").click(function () {
     $("#book_title").val($("#bookAuthor").val());
     $("#bookAuthor").val(title);
 });
+

cps/static/js/filter_grid.js

@@ -21,47 +21,59 @@ var $list = $("#list").isotope({
     itemSelector: ".book",
     layoutMode: "fitRows",
     getSortData: {
-        title: ".title",
+        title: ".title"
-    }
+    },
 });
 
+
 $("#desc").click(function() {
     if (direction === 0) {
         return;
     }
+    $("#asc").removeClass("active");
+    $("#desc").addClass("active");
+
     var page = $(this).data("id");
     $.ajax({
         method:"post",
         contentType: "application/json; charset=utf-8",
         dataType: "json",
-        url: window.location.pathname + "/../../ajax/view",
+        url: getPath() + "/ajax/view",
         data: "{\"" + page + "\": {\"dir\": \"desc\"}}",
     });
+    // invert sorting order to make already inverted start order working
     $list.isotope({
         sortBy: "name",
-        sortAscending: true
+        sortAscending: !$list.data('isotope').options.sortAscending
     });
+    direction = 0;
 });
 
 $("#asc").click(function() {
     if (direction === 1) {
         return;
     }
+    $("#desc").removeClass("active");
+    $("#asc").addClass("active");
+
     var page = $(this).data("id");
     $.ajax({
         method:"post",
         contentType: "application/json; charset=utf-8",
         dataType: "json",
-        url: window.location.pathname + "/../../ajax/view",
+        url: getPath() + "/ajax/view",
         data: "{\"" + page + "\": {\"dir\": \"asc\"}}",
     });
     $list.isotope({
         sortBy: "name",
-        sortAscending: false
+        sortAscending: !$list.data('isotope').options.sortAscending
     });
+    direction = 1;
 });
 
 $("#all").click(function() {
+    $(".char").removeClass("active");
+    $("#all").addClass("active");
     // go through all elements and make them visible
     $list.isotope({ filter: function() {
         return true;
@@ -70,6 +82,9 @@ $("#all").click(function() {
 });
 
 $(".char").click(function() {
+    $(".char").removeClass("active");
+    $(this).addClass("active");
+    $("#all").removeClass("active");
     var character = this.innerText;
     $list.isotope({ filter: function() {
         return this.attributes["data-id"].value.charAt(0).toUpperCase() === character;

cps/static/js/filter_list.js

@@ -19,6 +19,7 @@ var direction = $("#asc").data('order');  // 0=Descending order; 1= ascending or
 var sort = 0;       // Show sorted entries
 
 $("#sort_name").click(function() {
+    $("#sort_name").toggleClass("active");
     var className = $("h1").attr("Class") + "_sort_name";
     var obj = {};
     obj[className] = sort;
@@ -68,12 +69,15 @@ $("#desc").click(function() {
     if (direction === 0) {
         return;
     }
+    $("#asc").removeClass("active");
+    $("#desc").addClass("active");
+
     var page = $(this).data("id");
     $.ajax({
         method:"post",
         contentType: "application/json; charset=utf-8",
         dataType: "json",
-        url: window.location.pathname + "/../../ajax/view",
+        url: getPath() + "/ajax/view",
         data: "{\"" + page + "\": {\"dir\": \"desc\"}}",
     });
     var index = 0;
@@ -112,16 +116,18 @@ $("#desc").click(function() {
 
 
 $("#asc").click(function() {
-
     if (direction === 1) {
         return;
     }
+    $("#desc").removeClass("active");
+    $("#asc").addClass("active");
+
     var page = $(this).data("id");
     $.ajax({
         method:"post",
         contentType: "application/json; charset=utf-8",
         dataType: "json",
-        url: window.location.pathname + "/../../ajax/view",
+        url: getPath() + "/ajax/view",
         data: "{\"" + page + "\": {\"dir\": \"asc\"}}",
     });
     var index = 0;
@@ -159,6 +165,8 @@ $("#asc").click(function() {
 });
 
 $("#all").click(function() {
+    $("#all").addClass("active");
+    $(".char").removeClass("active");
     var cnt = $("#second").contents();
     $("#list").append(cnt);
     // Find count of middle element
@@ -176,6 +184,9 @@ $("#all").click(function() {
 });
 
 $(".char").click(function() {
+    $(".char").removeClass("active");
+    $(this).addClass("active");
+    $("#all").removeClass("active");
     var character = this.innerText;
     var count = 0;
     var index = 0;

cps/static/js/get_meta.js

@@ -14,37 +14,11 @@
  *  You should have received a copy of the GNU General Public License
  *  along with this program. If not, see <http://www.gnu.org/licenses/>.
  */
-/*
+/* global _, i18nMsg, tinymce, getPath */
- * Get Metadata from Douban Books api and Google Books api and ComicVine
- * Google Books api document: https://developers.google.com/books/docs/v1/using
- * Douban Books api document: https://developers.douban.com/wiki/?title=book_v2 (Chinese Only)
- * ComicVine api document: https://comicvine.gamespot.com/api/documentation
- */
-/* global _, i18nMsg, tinymce */
-var dbResults = [];
-var ggResults = [];
-var cvResults = [];
-var gsResults = [];
 
 $(function () {
     var msg = i18nMsg;
-  var douban = "https://api.douban.com";
+    var keyword = ""
-  var dbSearch = "/v2/book/search";
-  var dbDone = 0;
-
-  var google = "https://www.googleapis.com";
-  var ggSearch = "/books/v1/volumes";
-  var ggDone = 0;
-
-  var comicvine = "https://comicvine.gamespot.com";
-  var cvSearch = "/api/search/";
-  var cvDone = 0;
-
-  var googlescholar = window.location.href.split('/admin/book')[0];
-  var gsSearch = "/scholarsearch/"
-  var gsDone = 0;
-
-  var showFlag = 0;
 
     var templates = {
         bookResult: _.template(
@@ -54,17 +28,19 @@ $(function () {
 
     function populateForm (book) {
         tinymce.get("description").setContent(book.description);
-    var uniqueTags = [];
+        var uniqueTags = $.map($("#tags").val().split(","), $.trim);
+        if ( uniqueTags.length == 1 && uniqueTags[0] == "") {
+            uniqueTags = [];
+        }
         $.each(book.tags, function(i, el) {
             if ($.inArray(el, uniqueTags) === -1) uniqueTags.push(el);
         });
-
         var ampSeparatedAuthors = (book.authors || []).join(" & ");
         $("#bookAuthor").val(ampSeparatedAuthors);
         $("#book_title").val(book.title);
-    $("#tags").val(uniqueTags.join(","));
+        $("#tags").val(uniqueTags.join(", "));
         $("#rating").data("rating").setValue(Math.round(book.rating));
-    if(book.cover !== null){
+        if(book.cover && $("#cover_url").length){
             $(".cover img").attr("src", book.cover);
             $("#cover_url").val(book.cover);
         }
@@ -75,319 +51,98 @@ $(function () {
         }
     }
 
-  function showResult () {
+    function doSearch (keyword) {
-    showFlag++;
+        if (keyword) {
-    if (showFlag === 1) {
+            $("#meta-info").text(msg.loading);
-      $("#meta-info").html("<ul id=\"book-list\" class=\"media-list\"></ul>");
-    }
-    if ((ggDone === 3 || (ggDone === 1 && ggResults.length === 0)) &&
-      (dbDone === 3 || (dbDone === 1 && dbResults.length === 0)) &&
-      (cvDone === 3 || (cvDone === 1 && cvResults.length === 0)) &&
-      (gsDone === 3 || (gsDone === 1 && gsResults.length === 0))) {
-      $("#meta-info").html("<p class=\"text-danger\">" + msg.no_result + "</p>");
-      return;
-    }
-    function formatDate (date) {
-      var d = new Date(date),
-        month = "" + (d.getMonth() + 1),
-        day = "" + d.getDate(),
-        year = d.getFullYear();
-
-      if (month.length < 2) {
-        month = "0" + month;
-      }
-      if (day.length < 2) {
-        day = "0" + day;
-      }
-
-      return [year, month, day].join("-");
-    }
-    function generateID (title) {
-      return title.split("").reduce(function(a,b){a=((a<<5)-a)+b.charCodeAt(0);return a&a},0).toString().substr(0,12);
-    }
-
-    if (ggResults.length > 0) {
-      if (ggDone < 2) {
-        ggResults.forEach(function(result) {
-          var book = {
-            id: result.id,
-            title: result.volumeInfo.title,
-            authors: result.volumeInfo.authors || [],
-            description: result.volumeInfo.description || "",
-            publisher: result.volumeInfo.publisher || "",
-            publishedDate: result.volumeInfo.publishedDate || "",
-            tags: result.volumeInfo.categories || [],
-            rating: result.volumeInfo.averageRating || 0,
-            cover: result.volumeInfo.imageLinks ?
-            result.volumeInfo.imageLinks.thumbnail : location + "/../../../static/generic_cover.jpg",
-            url: "https://books.google.com/books?id=" + result.id,
-            source: {
-              id: "google",
-              description: "Google Books",
-              url: "https://books.google.com/"
-            }
-          };
-
-          var $book = $(templates.bookResult(book));
-          $book.find("img").on("click", function () {
-            populateForm(book);
-          });
-
-          $("#book-list").append($book);
-        });
-        ggDone = 2;
-      } else {
-        ggDone = 3;
-      }
-    }
-
-    if (gsResults.length > 0) {
-      if (gsDone < 2) {
-        gsResults.forEach(function(result) {
-          var book = {
-            id: generateID(result.bib.title),
-            title: result.bib.title,
-            authors: result.bib.author || [],
-            description: result.bib.abstract || "",
-            publisher: result.bib.venue || "",
-            publishedDate: result.bib.pub_year ? result.bib.pub_year+"-01-01" : "",
-            tags: [],
-            rating: 0,
-            series: "",
-            cover: null,
-            url: result.pub_url || result.eprint_url || "",
-            source: {
-              id: "googlescholar",
-              description: "Google Scholar",
-              link: "https://scholar.google.com/"
-            }
-          }
-          var $book = $(templates.bookResult(book));
-          $book.find("img").on("click", function () {
-            populateForm(book);
-          });
-
-          $("#book-list").append($book);
-
-        });
-        gsDone = 2;
-      }
-      else {
-        gsDone = 3;
-      }
-    }
-
-    if (dbResults.length > 0) {
-      if (dbDone < 2) {
-        dbResults.forEach(function(result) {
-          var seriesTitle = "";
-          if (result.series) {
-            seriesTitle = result.series.title;
-          }
-          var dateFomers = result.pubdate.split("-");
-          var publishedYear = parseInt(dateFomers[0], 10);
-          var publishedMonth = parseInt(dateFomers[1], 10);
-          var publishedDate = new Date(publishedYear, publishedMonth - 1, 1);
-
-          publishedDate = formatDate(publishedDate);
-
-          var book = {
-            id: result.id,
-            title: result.title,
-            authors: result.author || [],
-            description: result.summary,
-            publisher: result.publisher || "",
-            publishedDate: publishedDate || "",
-            tags: result.tags.map(function(tag) {
-              return tag.title.toLowerCase().replace(/,/g, "_");
-            }),
-            rating: result.rating.average || 0,
-            series: seriesTitle || "",
-            cover: result.image,
-            url: "https://book.douban.com/subject/" + result.id,
-            source: {
-              id: "douban",
-              description: "Douban Books",
-              url: "https://book.douban.com/"
-            }
-          };
-
-          if (book.rating > 0) {
-            book.rating /= 2;
-          }
-
-          var $book = $(templates.bookResult(book));
-          $book.find("img").on("click", function () {
-            populateForm(book);
-          });
-
-          $("#book-list").append($book);
-        });
-        dbDone = 2;
-      } else {
-        dbDone = 3;
-      }
-    }
-    if (cvResults.length > 0) {
-      if (cvDone < 2) {
-        cvResults.forEach(function(result) {
-          var seriesTitle = "";
-          if (result.volume.name) {
-            seriesTitle = result.volume.name;
-          }
-          var dateFomers = "";
-          if (result.store_date) {
-            dateFomers = result.store_date.split("-");
-          } else {
-            dateFomers = result.date_added.split("-");
-          }
-          var publishedYear = parseInt(dateFomers[0], 10);
-          var publishedMonth = parseInt(dateFomers[1], 10);
-          var publishedDate = new Date(publishedYear, publishedMonth - 1, 1);
-
-          publishedDate = formatDate(publishedDate);
-
-          var book = {
-            id: result.id,
-            title: seriesTitle + " #" + ("00" + result.issue_number).slice(-3) + " - " + result.name,
-            authors: result.author || [],
-            description: result.description,
-            publisher: "",
-            publishedDate: publishedDate || "",
-            tags: ["Comics", seriesTitle],
-            rating: 0,
-            series: seriesTitle || "",
-            cover: result.image.original_url,
-            url: result.site_detail_url,
-            source: {
-              id: "comicvine",
-              description: "ComicVine Books",
-              url: "https://comicvine.gamespot.com/"
-            }
-          };
-
-          var $book = $(templates.bookResult(book));
-          $book.find("img").on("click", function () {
-            populateForm(book);
-          });
-
-          $("#book-list").append($book);
-        });
-        cvDone = 2;
-      } else {
-        cvDone = 3;
-      }
-    }
-  }
-
-  function ggSearchBook (title) {
             $.ajax({
-      url: google + ggSearch + "?q=" + title.replace(/\s+/gm, "+"),
+                url: getPath() + "/metadata/search",
-      type: "GET",
+                type: "POST",
-      dataType: "jsonp",
+                data: {"query": keyword},
-      jsonp: "callback",
-      success: function success(data) {
-        if ("items" in data) {
-          ggResults = data.items;
-        }
-      },
-      complete: function complete() {
-        ggDone = 1;
-        showResult();
-        $("#show-google").trigger("change");
-      }
-    });
-  }
-
-  function dbSearchBook (title) {
-    var apikey = "054022eaeae0b00e0fc068c0c0a2102a";
-    $.ajax({
-      url: douban + dbSearch + "?apikey=" + apikey + "&q=" + title + "&fields=all&count=10",
-      type: "GET",
-      dataType: "jsonp",
-      jsonp: "callback",
-      success: function success(data) {
-        dbResults = data.books;
-      },
-      error: function error() {
-        $("#meta-info").html("<p class=\"text-danger\">" + msg.search_error + "!</p>" + $("#meta-info")[0].innerHTML);
-      },
-      complete: function complete() {
-        dbDone = 1;
-        showResult();
-        $("#show-douban").trigger("change");
-      }
-    });
-  }
-
-  function cvSearchBook (title) {
-    var apikey = "57558043c53943d5d1e96a9ad425b0eb85532ee6";
-    title = encodeURIComponent(title);
-    $.ajax({
-      url: comicvine + cvSearch + "?api_key=" + apikey + "&resources=issue&query=" + title + "&sort=name:desc&format=jsonp",
-      type: "GET",
-      dataType: "jsonp",
-      jsonp: "json_callback",
-      success: function success(data) {
-        cvResults = data.results;
-      },
-      error: function error() {
-        $("#meta-info").html("<p class=\"text-danger\">" + msg.search_error + "!</p>" + $("#meta-info")[0].innerHTML);
-      },
-      complete: function complete() {
-        cvDone = 1;
-        showResult();
-        $("#show-comics").trigger("change");
-      }
-    });
-  }
-
-  function gsSearchBook (title) {
-    $.ajax({
-      url: googlescholar + gsSearch + title.replace(/\s+/gm,'+'),
-      type: "GET",
                 dataType: "json",
                 success: function success(data) {
-        gsResults = data;
+                    $("#meta-info").html("<ul id=\"book-list\" class=\"media-list\"></ul>");
+                    data.forEach(function(book) {
+                        var $book = $(templates.bookResult(book));
+                        $book.find("img").on("click", function () {
+                            populateForm(book);
+                        });
+                        $("#book-list").append($book);
+                    });
                 },
-      complete: function complete() {
+                error: function error() {
-        gsDone = 1;
+                    $("#meta-info").html("<p class=\"text-danger\">" + msg.search_error + "!</p>" + $("#meta-info")[0].innerHTML);
-        showResult();
+                },
-        $("#show-googlescholar").trigger("change");
+            });
         }
-    })
     }
 
-  function doSearch (keyword) {
+    function populate_provider() {
-    showFlag = 0;
+        $("#metadata_provider").empty();
-    dbDone = ggDone = cvDone = 0;
+        $.ajax({
-    dbResults = [];
+            url: getPath() + "/metadata/provider",
-    ggResults = [];
+            type: "get",
-    cvResults = [];
+            dataType: "json",
-    gsResults = [];
+            success: function success(data) {
-    $("#meta-info").text(msg.loading);
+                data.forEach(function(provider) {
-    if (keyword) {
+                    var checked = "";
-      dbSearchBook(keyword);
+                    if (provider.active) {
-      ggSearchBook(keyword);
+                        checked = "checked";
-      cvSearchBook(keyword);
-      gsSearchBook(keyword);
                     }
+                    var $provider_button = '<input type="checkbox" id="show-' + provider.name + '" class="pill" data-initial="' + provider.initial + '" data-control="' + provider.id + '" ' + checked + '><label for="show-' + provider.name + '">' + provider.name + ' <span class="glyphicon glyphicon-ok"></span></label>'
+                    $("#metadata_provider").append($provider_button);
+                });
+            },
+        });
     }
 
+    $(document).on("change", ".pill", function () {
+        var element = $(this);
+        var id = element.data("control");
+        var initial = element.data("initial");
+        var val = element.prop('checked');
+        var params = {id : id, value: val};
+        if (!initial) {
+            params['initial'] = initial;
+            params['query'] = keyword;
+        }
+        $.ajax({
+            method:"post",
+            contentType: "application/json; charset=utf-8",
+            dataType: "json",
+            url: getPath() + "/metadata/provider/" + id,
+            data: JSON.stringify(params),
+            success: function success(data) {
+                element.data("initial", "true");
+                data.forEach(function(book) {
+                    var $book = $(templates.bookResult(book));
+                    $book.find("img").on("click", function () {
+                        populateForm(book);
+                    });
+                    $("#book-list").append($book);
+                });
+            }
+        });
+    });
+
     $("#meta-search").on("submit", function (e) {
         e.preventDefault();
-    var keyword = $("#keyword").val();
+        keyword = $("#keyword").val();
-    if (keyword) {
+        $('.pill').each(function(){
+            $(this).data("initial", $(this).prop('checked'));
+        });
         doSearch(keyword);
-    }
     });
 
     $("#get_meta").click(function () {
+        populate_provider();
         var bookTitle = $("#book_title").val();
-    if (bookTitle) {
         $("#keyword").val(bookTitle);
+        keyword = bookTitle;
         doSearch(bookTitle);
-    }
     });
-
+    $("#metaModal").on("show.bs.modal", function(e) {
+        $(e.relatedTarget).one('focus', function (e) {
+            $(this).blur();
+        });
+    });
 });

cps/static/js/kthoom.js

@@ -180,13 +180,15 @@ function initProgressClick() {
 
 function loadFromArrayBuffer(ab) {
     var lastCompletion = 0;
+    const collator = new Intl.Collator('en', { numeric: true, sensitivity: 'base' });
     loadArchiveFormats(['rar', 'zip', 'tar'], function() {
         // Open the file as an archive
         archiveOpenFile(ab, function (archive) {
             if (archive) {
                 totalImages = archive.entries.length
                 console.info('Uncompressing ' + archive.archive_type + ' ...');
-                archive.entries.forEach(function(e, i) {
+                entries = archive.entries.sort((a,b) => collator.compare(a.name, b.name));
+                entries.forEach(function(e, i) {
                     updateProgress( (i + 1)/ totalImages * 100);
                     if (e.is_file) {
                         e.readData(function(d) {

cps/static/js/libs/bootstrap-datepicker/locales/bootstrap-datepicker.ko.min.js

@@ -0,0 +1 @@
+!function(a){a.fn.datepicker.dates.ko={days:["일요일","월요일","화요일","수요일","목요일","금요일","토요일"],daysShort:["일","월","화","수","목","금","토"],daysMin:["일","월","화","수","목","금","토"],months:["1월","2월","3월","4월","5월","6월","7월","8월","9월","10월","11월","12월"],monthsShort:["1월","2월","3월","4월","5월","6월","7월","8월","9월","10월","11월","12월"],today:"오늘",clear:"삭제",format:"yyyy-mm-dd",titleFormat:"yyyy년mm월",weekStart:0}}(jQuery);

cps/static/js/libs/bootstrap-datepicker/locales/bootstrap-datepicker.zh_Hant_TW.min.js

@@ -0,0 +1,3 @@
+!function(a){a.fn.datepicker.dates["zh-TW"]={days:["星期日","星期一","星期二","星期三","星期四","星期五","星期六"],daysShort:["週日","週一","週二","週三","週四","週五","週六"],daysMin:["日","一","二","三","四","五","六"],months:["一月","二月","三月","四月","五月","六月","七月","八月","九月","十月","十一月","十二月"],monthsShort:["1月","2月","3月","4月","5月","6月","7月","8月","9月","10月","11月","12月"],today:"今天",format:"yyyy年mm月dd日",weekStart:1,clear:"清除"}}(jQuery);
+
+