Update Version

Update teststatus

MANIFEST.in

@@ -1 +1,3 @@
 graft src/calibreweb
+global-exclude __pycache__
+global-exclude *.pyc

README.md

@@ -19,13 +19,13 @@ Calibre-Web is a web app that offers a clean and intuitive interface for browsin
    - [Quick start](#quick-start)
    - [Requirements](#requirements)
 4. [Docker Images](#docker-images)
-5. [Contributor Recognition](#contributor-recognition)
-6. [Contact](#contact)
-7. [Contributing to Calibre-Web](#contributing-to-calibre-web)
+5. [Troubleshooting](#troubleshooting)
+6. [Contributor Recognition](#contributor-recognition)
+7. [Contact](#contact)
+8. [Contributing to Calibre-Web](#contributing-to-calibre-web)
 
 </details>
 
-
 *This software is a fork of [library](https://github.com/mutschler/calibreserver) and licensed under the GPL v3 License.*
 
 ![Main screen](https://github.com/janeczku/calibre-web/wiki/images/main_screen.png)
@@ -57,52 +57,103 @@ Calibre-Web is a web app that offers a clean and intuitive interface for browsin
 
 ## Installation
 
-#### Installation via pip (recommended)
-1. Create a virtual environment for Calibre-Web to avoid conflicts with existing Python dependencies
-2. Install Calibre-Web via pip: `pip install calibreweb` (or `pip3` depending on your OS/distro)
-3. Install optional features via pip as needed, see [this page](https://github.com/janeczku/calibre-web/wiki/Dependencies-in-Calibre-Web-Linux-and-Windows) for details
-4. Start Calibre-Web by typing `cps`
+### Installation via pip (recommended)
 
-*Note: Raspberry Pi OS users may encounter issues during installation. If so, please update pip (`./venv/bin/python3 -m pip install --upgrade pip`) and/or install cargo (`sudo apt install cargo`) before retrying the installation.*
+1. **Create a virtual environment**: It’s essential to isolate your Calibre-Web installation to avoid dependency conflicts. You can create a virtual environment by running:
+   ```
+   python3 -m venv calibre-web-env
+   ```
+2. **Activate the virtual environment**:
+   ```
+   source calibre-web-env/bin/activate
+   ```
+3. **Install Calibre-Web**: Use pip to install the application:
+   ```
+   pip install calibreweb
+   ```
+4. **Install optional features**: For additional functionality, you may need to install optional features. Refer to [this page](https://github.com/janeczku/calibre-web/wiki/Dependencies-in-Calibre-Web-Linux-and-Windows) for details on what can be installed.
+5. **Start Calibre-Web**: After installation, you can start the application with:
+   ```
+   cps
+   ```
 
-Refer to the Wiki for additional installation examples: [manual installation](https://github.com/janeczku/calibre-web/wiki/Manual-installation), [Linux Mint](https://github.com/janeczku/calibre-web/wiki/How-To:-Install-Calibre-Web-in-Linux-Mint-19-or-20), [Cloud Provider](https://github.com/janeczku/calibre-web/wiki/How-To:-Install-Calibre-Web-on-a-Cloud-Provider).
+*Note: Users of Raspberry Pi OS may encounter installation issues. If you do, try upgrading pip and/or installing cargo as follows:*
+   ```
+   ./venv/bin/python3 -m pip install --upgrade pip
+   sudo apt install cargo
+   ```
+
+### Important Links
+- For additional installation examples, check the following:
+   - [Manual installation](https://github.com/janeczku/calibre-web/wiki/Manual-installation)
+   - [Linux Mint installation](https://github.com/janeczku/calibre-web/wiki/How-To:-Install-Calibre-Web-in-Linux-Mint-19-or-20)
+   - [Cloud Provider setup](https://github.com/janeczku/calibre-web/wiki/How-To:-Install-Calibre-Web-on-a-Cloud-Provider)
 
 ## Quick Start
 
-1. Open your browser and navigate to `http://localhost:8083` or `http://localhost:8083/opds` for the OPDS catalog
-2. Log in with the default admin credentials
-3. If you don't have a Calibre database, you can use [this database](https://github.com/janeczku/calibre-web/raw/master/library/metadata.db) (move it out of the Calibre-Web folder to prevent overwriting during updates)
-4. Set `Location of Calibre database` to the path of the folder containing your Calibre library (metadata.db) and click "Save"
-5. Optionally, use Google Drive to host your Calibre library by following the [Google Drive integration guide](https://github.com/janeczku/calibre-web/wiki/G-Drive-Setup#using-google-drive-integration)
-6. Configure your Calibre-Web instance via the admin page, referring to the [Basic Configuration](https://github.com/janeczku/calibre-web/wiki/Configuration#basic-configuration) and [UI Configuration](https://github.com/janeczku/calibre-web/wiki/Configuration#ui-configuration) guides
-
-#### Default Admin Login:
-- **Username:** admin
-- **Password:** admin123
+1. **Access Calibre-Web**: Open your browser and navigate to:
+   ```
+   http://localhost:8083
+   ```
+   or for the OPDS catalog:
+   ```
+   http://localhost:8083/opds
+   ```
+2. **Log in**: Use the default admin credentials:
+   - **Username:** admin
+   - **Password:** admin123
+3. **Database Setup**: If you do not have a Calibre database, download a sample from:
+   ```
+   https://github.com/janeczku/calibre-web/raw/master/library/metadata.db
+   ```
+   Move it out of the Calibre-Web folder to avoid overwriting during updates.
+4. **Configure Calibre Database**: In the admin interface, set the `Location of Calibre database` to the path of the folder containing your Calibre library (where `metadata.db` is located) and click "Save".
+5. **Google Drive Integration**: For hosting your Calibre library on Google Drive, refer to the [Google Drive integration guide](https://github.com/janeczku/calibre-web/wiki/G-Drive-Setup#using-google-drive-integration).
+6. **Admin Configuration**: Configure your instance via the admin page, referring to the [Basic Configuration](https://github.com/janeczku/calibre-web/wiki/Configuration#basic-configuration) and [UI Configuration](https://github.com/janeczku/calibre-web/wiki/Configuration#ui-configuration) guides.
 
 ## Requirements
 
-- Python 3.7+
-- [Imagemagick](https://imagemagick.org/script/download.php) for cover extraction from EPUBs (Windows users may need to install [Ghostscript](https://ghostscript.com/releases/gsdnld.html) for PDF cover extraction)
-- Windows users need to install [libmagic for 32bit python](https://gnuwin32.sourceforge.net/downlinks/file.php) or [libmagic for 64bit python](https://github.com/nscaife/file-windows/releases/tag/20170108), depending on the python version; The files need to be installed in path (e.g. script folder of your Calibre-Web venv, or in the root folder of Calibre-Web
-- Optional: [Calibre desktop program](https://calibre-ebook.com/download) for on-the-fly conversion and metadata editing (set "calibre's converter tool" path on the setup page)
-- Optional: [Kepubify tool](https://github.com/pgaskin/kepubify/releases/latest) for Kobo device support (place the binary in `/opt/kepubify` on Linux or `C:\Program Files\kepubify` on Windows)
+- **Python Version**: Ensure you have Python 3.7 or newer.
+- **Imagemagick**: Required for cover extraction from EPUBs. Windows users may also need to install [Ghostscript](https://ghostscript.com/releases/gsdnld.html) for PDF cover extraction.
+- **Windows Users**: Install [libmagic](https://gnuwin32.sourceforge.net/downlinks/file.php) for 32bit python or [libmagic for 64bit python](https://github.com/nscaife/file-windows/releases/tag/20170108), depending on your Python version. Make sure these files are included in your path.
+- **Optional Tools**:
+   - **Calibre desktop program**: Recommended for on-the-fly conversion and metadata editing. Set the path to Calibre’s converter tool on the setup page.
+   - **Kepubify tool**: Needed for Kobo device support. Download the tool and place the binary in `/opt/kepubify` on Linux or `C:\Program Files\kepubify` on Windows.
 
 ## Docker Images
 
-Pre-built Docker images are available in the following Docker Hub repositories (maintained by the LinuxServer team):
+Pre-built Docker images are available:
 
-#### **LinuxServer - x64, aarch64**
-- [Docker Hub](https://hub.docker.com/r/linuxserver/calibre-web)
-- [GitHub](https://github.com/linuxserver/docker-calibre-web)
-- [GitHub - Optional Calibre layer](https://github.com/linuxserver/docker-mods/tree/universal-calibre)
+### **LinuxServer - x64, aarch64**
+- **Docker Hub**: [linuxserver/calibre-web](https://hub.docker.com/r/linuxserver/calibre-web)
+- **GitHub**: [linuxserver/docker-calibre-web](https://github.com/linuxserver/docker-calibre-web)
+- **Optional Calibre layer**: [linuxserver/docker-mods](https://github.com/linuxserver/docker-mods/tree/universal-calibre)
 
-  Include the environment variable `DOCKER_MODS=linuxserver/mods:universal-calibre` in your Docker run/compose file to add the Calibre `ebook-convert` binary (x64 only). Omit this variable for a lightweight image.
+To include the Calibre `ebook-convert` binary (x64 only), add the environment variable:
+``` 
+DOCKER_MODS=linuxserver/mods:universal-calibre
+```
+in your Docker run/compose file. Omit this variable for a lightweight image.
 
-  Both the Calibre-Web and Calibre-Mod images are automatically rebuilt on new releases and updates.
+- **Paths Configuration**:
+   - Set **Path to Calibre Binaries** to `/usr/bin`.
+   - Set **Path to Unrar** to `/usr/bin/unrar`.
 
-  - Set "Path to Calibre Binaries" to `/usr/bin`
-  - Set "Path to Unrar" to `/usr/bin/unrar`
+## Troubleshooting
+
+- **Common Issues**: 
+   - If you experience issues starting the application, check the log files located in the `logs` directory for error messages.
+   - If eBooks fail to load, verify that the `Location of Calibre database` is correctly set and that the database file is accessible.
+
+- **Configuration Errors**: Ensure that your Calibre database is compatible and properly formatted. Refer to the Calibre documentation for guidance on maintaining the database.
+
+- **Performance Problems**: 
+   - If the application is slow, consider increasing the allocated resources (CPU/RAM) to your server or optimizing the Calibre database by removing duplicates and unnecessary entries.
+   - Regularly clear the cache in your web browser to improve loading times.
+
+- **User Management Issues**: If users are unable to log in or register, check the user permission settings in the admin interface. Ensure that registration is enabled and that users are being assigned appropriate roles.
+
+- **Support Resources**: For additional help, consider visiting the [FAQ section](https://github.com/janeczku/calibre-web/wiki/FAQ) of the wiki or posting your questions in the [Discord community](https://discord.gg/h2VsJ2NEfB).
 
 ## Contributor Recognition
 
@@ -116,4 +167,21 @@ For more information, How To's, and FAQs, please visit the [Wiki](https://github
 
 ## Contributing to Calibre-Web
 
-Check out our [Contributing Guidelines](https://github.com/janeczku/calibre-web/blob/master/CONTRIBUTING.md)
+To contribute, please check our [Contributing Guidelines](https://github.com/janeczku/calibre-web/blob/master/CONTRIBUTING.md). We welcome issues, feature requests, and pull requests from the community.
+
+### Reporting Bugs
+
+If you encounter bugs or issues, please report them in the [issues section](https://github.com/janeczku/calibre-web/issues) of the repository. Be sure to include detailed information about your setup and the problem encountered.
+
+### Feature Requests
+
+We welcome suggestions for new features. Please create a new issue in the repository to discuss your ideas.
+
+## Additional Resources
+
+- **Documentation**: Comprehensive documentation is available on the [Calibre-Web wiki](https://github.com/janeczku/calibre-web/wiki).
+- **Community Contributions**: Explore the [community contributions](https://github.com/janeczku/calibre-web/pulls) to see ongoing work and how you can get involved.
+
+---
+
+Thank you for using Calibre-Web! We hope you enjoy managing your eBook library with our tool.

cps/__init__.py

@@ -31,12 +31,10 @@ from flask_principal import Principal
 
 from . import logger
 from .cli import CliParameter
-from .constants import CONFIG_DIR
 from .reverseproxy import ReverseProxied
 from .server import WebServer
 from .dep_check import dependency_check
 from .updater import Updater
-from .babel import babel, get_locale
 from . import config_sql
 from . import cache_buster
 from . import ub, db
@@ -86,7 +84,7 @@ log = logger.create()
 app = Flask(__name__)
 app.config.update(
     SESSION_COOKIE_HTTPONLY=True,
-    SESSION_COOKIE_SAMESITE='Strict',
+    SESSION_COOKIE_SAMESITE='Lax',
     REMEMBER_COOKIE_SAMESITE='Strict',
     WTF_CSRF_SSL_STRICT=False,
     SESSION_COOKIE_NAME=os.environ.get('COOKIE_PREFIX', "") + "session",
@@ -183,6 +181,7 @@ def create_app():
     app.secret_key = os.getenv('SECRET_KEY', config_sql.get_flask_session_key(ub.session))
 
     web_server.init_app(app, config)
+    from .cw_babel import babel, get_locale
     if hasattr(babel, "localeselector"):
         babel.init_app(app)
         babel.localeselector(get_locale)

cps/about.py

@@ -42,19 +42,17 @@ opt = dep_check.load_dependencies(True)
 for i in (req + opt):
     modules[i[1]] = i[0]
 modules['Jinja2'] = importlib.metadata.version("jinja2")
-try:
+if sys.version_info < (3, 12):
     modules['pySqlite'] = sqlite3.version
-except Exception:
-    pass
 modules['SQLite'] = sqlite3.sqlite_version
 sorted_modules = OrderedDict((sorted(modules.items(), key=lambda x: x[0].casefold())))
 
 
 def collect_stats():
     if constants.NIGHTLY_VERSION[0] == "$Format:%H$":
-        calibre_web_version = constants.STABLE_VERSION['version'].replace("b", " Beta")
+        calibre_web_version = constants.STABLE_VERSION.replace("b", " Beta")
     else:
-        calibre_web_version = (constants.STABLE_VERSION['version'].replace("b", " Beta") + ' - '
+        calibre_web_version = (constants.STABLE_VERSION.replace("b", " Beta") + ' - '
                                + constants.NIGHTLY_VERSION[0].replace('%', '%%') + ' - '
                                + constants.NIGHTLY_VERSION[1].replace('%', '%%'))
 

cps/admin.py

@@ -52,7 +52,7 @@ from .gdriveutils import is_gdrive_ready, gdrive_support
 from .render_template import render_title_template, get_sidebar_config
 from .services.worker import WorkerThread
 from .usermanagement import user_login_required
-from .babel import get_available_translations, get_available_locale, get_user_locale_language
+from .cw_babel import get_available_translations, get_available_locale, get_user_locale_language
 from . import debug_info
 from .string_helper import strip_whitespaces
 
@@ -306,7 +306,15 @@ def edit_user_table():
         .group_by(text('books_tags_link.tag')) \
         .order_by(db.Tags.name).all()
     if config.config_restricted_column:
-        custom_values = calibre_db.session.query(db.cc_classes[config.config_restricted_column]).all()
+        try:
+            custom_values = calibre_db.session.query(db.cc_classes[config.config_restricted_column]).all()
+        except (KeyError, AttributeError, IndexError):
+            custom_values = []
+            log.error("Custom Column No.{} does not exist in calibre database".format(
+                config.config_restricted_column))
+            flash(_("Custom Column No.%(column)d does not exist in calibre database",
+                    column=config.config_restricted_column),
+                  category="error")
     else:
         custom_values = []
     if not config.config_anonbrowse:
@@ -982,8 +990,14 @@ def prepare_tags(user, action, tags_name, id_list):
             raise Exception(_("Tag not found"))
         new_tags_list = [x.name for x in tags]
     else:
-        tags = calibre_db.session.query(db.cc_classes[config.config_restricted_column]) \
-            .filter(db.cc_classes[config.config_restricted_column].id.in_(id_list)).all()
+        try:
+            tags = calibre_db.session.query(db.cc_classes[config.config_restricted_column]) \
+                .filter(db.cc_classes[config.config_restricted_column].id.in_(id_list)).all()
+        except (KeyError, AttributeError, IndexError):
+            log.error("Custom Column No.{} does not exist in calibre database".format(
+                config.config_restricted_column))
+            raise Exception(_("Custom Column No.%(column)d does not exist in calibre database",
+                    column=config.config_restricted_column))
         new_tags_list = [x.value for x in tags]
     saved_tags_list = user.__dict__[tags_name].split(",") if len(user.__dict__[tags_name]) else []
     if action == "remove":
@@ -1744,6 +1758,14 @@ def _db_configuration_update_helper():
             ub.session.query(ub.KoboSyncedBooks).delete()
             helper.delete_thumbnail_cache()
             ub.session_commit()
+            # deleted visibilities based on custom column and tags
+            config.config_restricted_column = 0
+            config.config_denied_tags = ""
+            config.config_allowed_tags = ""
+            config.config_columns_to_ignore = ""
+            config.config_denied_column_value = ""
+            config.config_allowed_column_value = ""
+            config.config_read_column = 0
         _config_string(to_save, "config_calibre_dir")
         calibre_db.update_config(config)
         if not os.access(os.path.join(config.config_calibre_dir, "metadata.db"), os.W_OK):
@@ -2081,7 +2103,7 @@ def _handle_edit_user(to_save, content, languages, translations, kobo_support):
 
 
 def extract_user_data_from_field(user, field):
-    match = re.search(field + r"=([@\.\d\s\w-]+)", user, re.IGNORECASE | re.UNICODE)
+    match = re.search(field + r"=(.*?)($|(?<!\\),)", user, re.IGNORECASE | re.UNICODE)    
     if match:
         return match.group(1)
     else:
@@ -2089,7 +2111,7 @@ def extract_user_data_from_field(user, field):
 
 
 def extract_dynamic_field_from_filter(user, filtr):
-    match = re.search("([a-zA-Z0-9-]+)=%s", filtr, re.IGNORECASE | re.UNICODE)
+    match = re.search(r"([a-zA-Z0-9-]+)=%s", filtr, re.IGNORECASE | re.UNICODE)
     if match:
         return match.group(1)
     else:

cps/basic.py

@@ -1,90 +0,0 @@
-#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
-#    Copyright (C) 2018-2019 OzzieIsaacs, cervinko, jkrehm, bodybybuddha, ok11,
-#                            andy29485, idalin, Kyosfonica, wuqi, Kennyl, lemmsh,
-#                            falgh1, grunjol, csitko, ytils, xybydy, trasba, vrabe,
-#                            ruben-herold, marblepebble, JackED42, SiphonSquirrel,
-#                            apetresc, nanu-c, mutschler, carderne
-#
-#  This program is free software: you can redistribute it and/or modify
-#  it under the terms of the GNU General Public License as published by
-#  the Free Software Foundation, either version 3 of the License, or
-#  (at your option) any later version.
-#
-#  This program is distributed in the hope that it will be useful,
-#  but WITHOUT ANY WARRANTY; without even the implied warranty of
-#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#  GNU General Public License for more details.
-#
-#  You should have received a copy of the GNU General Public License
-#  along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-
-from cps.pagination import Pagination
-from flask import Blueprint
-from flask_babel import gettext as _
-from flask_babel import get_locale
-from flask import request, redirect, url_for
-
-from . import logger, isoLanguages
-from . import db, config
-from . import calibre_db
-from .usermanagement import login_required_if_no_ano
-from .render_template import render_title_template
-from .web import get_sort_function
-
-try:
-    from natsort import natsorted as sort
-except ImportError:
-    sort = sorted  # Just use regular sort then, may cause issues with badly named pages in cbz/cbr files
-
-basic = Blueprint('basic', __name__)
-
-log = logger.create()
-
-
-@basic.route("/basic", methods=["GET"])
-@login_required_if_no_ano
-def index():
-    term = request.args.get("query", "")  # default to showing all books
-    limit = 15
-    page = int(request.args.get("page") or 1)
-    off = (page - 1) * limit
-    order = get_sort_function("stored", "search")
-    join = db.books_series_link, db.Books.id == db.books_series_link.c.book, db.Series
-    entries, result_count, pagination = calibre_db.get_search_results(term,
-                                                                      config,
-                                                                      off,
-                                                                      order,
-                                                                      limit,
-                                                                      *join)
-    return render_title_template('basic_index.html',
-                                 searchterm=term,
-                                 pagination=pagination,
-                                 query=term,
-                                 adv_searchterm=term,
-                                 entries=entries,
-                                 result_count=result_count,
-                                 title=_("Search"),
-                                 page="search",
-                                 order=order[1])
-
-
-@basic.route("/basic_book/<int:book_id>")
-@login_required_if_no_ano
-def show_book(book_id):
-    entries = calibre_db.get_book_read_archived(book_id, config.config_read_column, allow_show_archived=True)
-    if entries:
-        entry = entries[0]
-        for lang_index in range(0, len(entry.languages)):
-            entry.languages[lang_index].language_name = isoLanguages.get_language_name(get_locale(), entry.languages[
-                lang_index].lang_code)
-        entry.ordered_authors = calibre_db.order_authors([entry])
-
-        return render_title_template('basic_detail.html',
-                                     entry=entry,
-                                     is_xhr=request.headers.get('X-Requested-With') == 'XMLHttpRequest',
-                                     title=entry.title,
-                                     page="book")
-    else:
-        log.debug("Selected book is unavailable. File does not exist or is not accessible")
-        return redirect(url_for("basic.index"))

cps/cli.py

@@ -29,8 +29,8 @@ from .constants import DEFAULT_SETTINGS_FILE, DEFAULT_GDRIVE_FILE
 
 def version_info():
     if _NIGHTLY_VERSION[1].startswith('$Format'):
-        return "Calibre-Web version: %s - unknown git-clone" % _STABLE_VERSION['version'].replace("b", " Beta")
-    return "Calibre-Web version: %s -%s" % (_STABLE_VERSION['version'].replace("b", " Beta"), _NIGHTLY_VERSION[1])
+        return "Calibre-Web version: %s - unknown git-clone" % _STABLE_VERSION.replace("b", " Beta")
+    return "Calibre-Web version: %s -%s" % (_STABLE_VERSION.replace("b", " Beta"), _NIGHTLY_VERSION[1])
 
 
 class CliParameter(object):

cps/constants.py

@@ -19,9 +19,6 @@
 import sys
 import os
 from collections import namedtuple
-from sqlalchemy import __version__ as sql_version
-
-sqlalchemy_version2 = ([int(x) for x in sql_version.split('.')] >= [2, 0, 0])
 
 # APP_MODE - production, development, or test
 APP_MODE             = os.environ.get('APP_MODE', 'production')
@@ -175,7 +172,7 @@ BookMeta = namedtuple('BookMeta', 'file_path, extension, title, author, cover, d
                                   'series_id, languages, publisher, pubdate, identifiers')
 
 # python build process likes to have x.y.zbw -> b for beta and w a counting number
-STABLE_VERSION = {'version': '0.6.24b'}
+STABLE_VERSION =  '0.6.24'
 
 NIGHTLY_VERSION = dict()
 NIGHTLY_VERSION[0] = '$Format:%H$'

cps/cw_advocate/__init__.py

@@ -0,0 +1,22 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+
+from .adapters import ValidatingHTTPAdapter
+from .api import *
+from .addrvalidator import AddrValidator
+from .exceptions import UnacceptableAddressException

cps/cw_advocate/adapters.py

@@ -0,0 +1,48 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+from requests.adapters import HTTPAdapter, DEFAULT_POOLBLOCK
+
+from .addrvalidator import AddrValidator
+from .exceptions import ProxyDisabledException
+from .poolmanager import ValidatingPoolManager
+
+
+class ValidatingHTTPAdapter(HTTPAdapter):
+    __attrs__ = HTTPAdapter.__attrs__ + ['_validator']
+
+    def __init__(self, *args, **kwargs):
+        self._validator = kwargs.pop('validator', None)
+        if not self._validator:
+            self._validator = AddrValidator()
+        super().__init__(*args, **kwargs)
+
+    def init_poolmanager(self, connections, maxsize, block=DEFAULT_POOLBLOCK,
+                         **pool_kwargs):
+        self._pool_connections = connections
+        self._pool_maxsize = maxsize
+        self._pool_block = block
+        self.poolmanager = ValidatingPoolManager(
+            num_pools=connections,
+            maxsize=maxsize,
+            block=block,
+            validator=self._validator,
+            **pool_kwargs
+        )
+
+    def proxy_manager_for(self, proxy, **proxy_kwargs):
+        raise ProxyDisabledException("Proxies cannot be used with Advocate")

cps/cw_advocate/addrvalidator.py

@@ -0,0 +1,281 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+import functools
+import fnmatch
+import ipaddress
+import re
+
+try:
+    import netifaces
+    HAVE_NETIFACES = True
+except ImportError:
+    netifaces = None
+    HAVE_NETIFACES = False
+
+from .exceptions import NameserverException, ConfigException
+
+
+def canonicalize_hostname(hostname):
+    """Lowercase and punycodify a hostname"""
+    # We do the lowercasing after IDNA encoding because we only want to
+    # lowercase the *ASCII* chars.
+    # TODO: The differences between IDNA2003 and IDNA2008 might be relevant
+    # to us, but both specs are damn confusing.
+    return str(hostname.encode("idna").lower(), 'utf-8')
+
+
+def determine_local_addresses():
+    """Get all IPs that refer to this machine according to netifaces"""
+    if not HAVE_NETIFACES:
+        raise ConfigException("Tried to determine local addresses, "
+                              "but netifaces module was not importable")
+    ips = []
+    for interface in netifaces.interfaces():
+        if_families = netifaces.ifaddresses(interface)
+        for family_kind in {netifaces.AF_INET, netifaces.AF_INET6}:
+            addrs = if_families.get(family_kind, [])
+            for addr in (x.get("addr", "") for x in addrs):
+                if family_kind == netifaces.AF_INET6:
+                    # We can't do anything sensible with the scope here
+                    addr = addr.split("%")[0]
+                ips.append(ipaddress.ip_network(addr))
+    return ips
+
+
+def add_local_address_arg(func):
+    """Add the "_local_addresses" kwarg if it's missing
+
+    IMO this information shouldn't be cached between calls (what if one of the
+    adapters got a new IP at runtime?,) and we don't want each function to
+    recalculate it. Just recalculate it if the caller didn't provide it for us.
+    """
+    @functools.wraps(func)
+    def wrapper(self, *args, **kwargs):
+        if "_local_addresses" not in kwargs:
+            if self.autodetect_local_addresses:
+                kwargs["_local_addresses"] = determine_local_addresses()
+            else:
+                kwargs["_local_addresses"] = []
+        return func(self, *args, **kwargs)
+    return wrapper
+
+
+class AddrValidator:
+    _6TO4_RELAY_NET = ipaddress.ip_network("192.88.99.0/24")
+    # Just the well known prefix, DNS64 servers can set their own
+    # prefix, but in practice most probably don't.
+    _DNS64_WK_PREFIX = ipaddress.ip_network("64:ff9b::/96")
+    DEFAULT_PORT_WHITELIST = {80, 8080, 443, 8443, 8000}
+
+    def __init__(
+            self,
+            ip_blacklist=None,
+            ip_whitelist=None,
+            port_whitelist=None,
+            port_blacklist=None,
+            hostname_blacklist=None,
+            allow_ipv6=False,
+            allow_teredo=False,
+            allow_6to4=False,
+            allow_dns64=False,
+            # Must be explicitly set to "False" if you don't want to try
+            # detecting local interface addresses with netifaces.
+            autodetect_local_addresses=True,
+    ):
+        if not port_blacklist and not port_whitelist:
+            # An assortment of common HTTPS? ports.
+            port_whitelist = self.DEFAULT_PORT_WHITELIST.copy()
+        self.ip_blacklist = ip_blacklist or set()
+        self.ip_whitelist = ip_whitelist or set()
+        self.port_blacklist = port_blacklist or set()
+        self.port_whitelist = port_whitelist or set()
+        # TODO: ATM this can contain either regexes or globs that are converted
+        # to regexes upon every check. Create a collection that automagically
+        # converts them to regexes on insert?
+        self.hostname_blacklist = hostname_blacklist or set()
+        self.allow_ipv6 = allow_ipv6
+        self.allow_teredo = allow_teredo
+        self.allow_6to4 = allow_6to4
+        self.allow_dns64 = allow_dns64
+        self.autodetect_local_addresses = autodetect_local_addresses
+
+    @add_local_address_arg
+    def is_ip_allowed(self, addr_ip, _local_addresses=None):
+        if not isinstance(addr_ip,
+                          (ipaddress.IPv4Address, ipaddress.IPv6Address)):
+            addr_ip = ipaddress.ip_address(addr_ip)
+
+        # The whitelist should take precedence over the blacklist so we can
+        # punch holes in blacklisted ranges
+        if any(addr_ip in net for net in self.ip_whitelist):
+            return True
+
+        if any(addr_ip in net for net in self.ip_blacklist):
+            return False
+
+        if any(addr_ip in net for net in _local_addresses):
+            return False
+
+        if addr_ip.version == 4:
+            if not addr_ip.is_private:
+                # IPs for carrier-grade NAT. Seems weird that it doesn't set
+                # `is_private`, but we need to check `not is_global`
+                if not ipaddress.ip_network(addr_ip).is_global:
+                    return False
+        elif addr_ip.version == 6:
+            # You'd better have a good reason for enabling IPv6
+            # because Advocate's techniques don't work well without NAT.
+            if not self.allow_ipv6:
+                return False
+
+            # v6 addresses can also map to IPv4 addresses! Tricky!
+            v4_nested = []
+            if addr_ip.ipv4_mapped:
+                v4_nested.append(addr_ip.ipv4_mapped)
+            # WTF IPv6? Why you gotta have a billion tunneling mechanisms?
+            # XXX: Do we even really care about these? If we're tunneling
+            # through public servers we shouldn't be able to access
+            # addresses on our private network, right?
+            if addr_ip.sixtofour:
+                if not self.allow_6to4:
+                    return False
+                v4_nested.append(addr_ip.sixtofour)
+            if addr_ip.teredo:
+                if not self.allow_teredo:
+                    return False
+                # Check both the client *and* server IPs
+                v4_nested.extend(addr_ip.teredo)
+            if addr_ip in self._DNS64_WK_PREFIX:
+                if not self.allow_dns64:
+                    return False
+                # When using the well-known prefix the last 4 bytes
+                # are the IPv4 addr
+                v4_nested.append(ipaddress.ip_address(addr_ip.packed[-4:]))
+
+            if not all(self.is_ip_allowed(addr_v4) for addr_v4 in v4_nested):
+                return False
+
+            # fec0::*, apparently deprecated?
+            if addr_ip.is_site_local:
+                return False
+        else:
+            raise ValueError("Unsupported IP version(?): %r" % addr_ip)
+
+        # 169.254.XXX.XXX, AWS uses these for autoconfiguration
+        if addr_ip.is_link_local:
+            return False
+        # 127.0.0.1, ::1, etc.
+        if addr_ip.is_loopback:
+            return False
+        if addr_ip.is_multicast:
+            return False
+        # 192.168.XXX.XXX, 10.XXX.XXX.XXX
+        if addr_ip.is_private:
+            return False
+        # 255.255.255.255, ::ffff:XXXX:XXXX (v6->v4) mapping
+        if addr_ip.is_reserved:
+            return False
+        # There's no reason to connect directly to a 6to4 relay
+        if addr_ip in self._6TO4_RELAY_NET:
+            return False
+        # 0.0.0.0
+        if addr_ip.is_unspecified:
+            return False
+
+        # It doesn't look bad, so... it's must be ok!
+        return True
+
+    def _hostname_matches_pattern(self, hostname, pattern):
+        # If they specified a string, just assume they only want basic globbing.
+        # This stops people from not realizing they're dealing in REs and
+        # not escaping their periods unless they specifically pass in an RE.
+        # This has the added benefit of letting us sanely handle globbed
+        # IDNs by default.
+        if isinstance(pattern, str):
+            # convert the glob to a punycode glob, then a regex
+            pattern = fnmatch.translate(canonicalize_hostname(pattern))
+
+        hostname = canonicalize_hostname(hostname)
+        # Down the line the hostname may get treated as a null-terminated string
+        # (as with `socket.getaddrinfo`.) Try to account for that.
+        #
+        #    >>> socket.getaddrinfo("example.com\x00aaaa", 80)
+        #    [(2, 1, 6, '', ('93.184.216.34', 80)), [...]
+        no_null_hostname = hostname.split("\x00")[0]
+
+        return any(re.match(pattern, x.strip(".")) for x
+                   in (no_null_hostname, hostname))
+
+    def is_hostname_allowed(self, hostname):
+        # Sometimes (like with "external" services that your IP has privileged
+        # access to) you might not always know the IP range to blacklist access
+        # to, or the `A` record might change without you noticing.
+        # For e.x.: `foocorp.external.org`.
+        #
+        # Another option is doing something like:
+        #
+        #     for addrinfo in socket.getaddrinfo("foocorp.external.org", 80):
+        #         global_validator.ip_blacklist.add(ip_address(addrinfo[4][0]))
+        #
+        # but that's not always a good idea if they're behind a third-party lb.
+        for pattern in self.hostname_blacklist:
+            if self._hostname_matches_pattern(hostname, pattern):
+                return False
+        return True
+
+    @add_local_address_arg
+    def is_addrinfo_allowed(self, addrinfo, _local_addresses=None):
+        assert(len(addrinfo) == 5)
+        # XXX: Do we care about any of the other elements? Guessing not.
+        family, socktype, proto, canonname, sockaddr = addrinfo
+
+        # The 4th elem inaddrinfo may either be a touple of two or four items,
+        # depending on whether we're dealing with IPv4 or v6
+        if len(sockaddr) == 2:
+            # v4
+            ip, port = sockaddr
+        elif len(sockaddr) == 4:
+            # v6
+            # XXX: what *are* `flow_info` and `scope_id`? Anything useful?
+            # Seems like we can figure out all we need about the scope from
+            # the `is_<x>` properties.
+            ip, port, flow_info, scope_id = sockaddr
+        else:
+            raise ValueError("Unexpected addrinfo format %r" % sockaddr)
+
+        # Probably won't help protect against SSRF, but might prevent our being
+        # used to attack others' non-HTTP services. See
+        # http://www.remote.org/jochen/sec/hfpa/
+        if self.port_whitelist and port not in self.port_whitelist:
+            return False
+        if port in self.port_blacklist:
+            return False
+
+        if self.hostname_blacklist:
+            if not canonname:
+                raise NameserverException(
+                    "addrinfo must contain the canon name to do blacklisting "
+                    "based on hostname. Make sure you use the "
+                    "`socket.AI_CANONNAME` flag, and that each record contains "
+                    "the canon name. Your DNS server might also be garbage."
+                )
+
+            if not self.is_hostname_allowed(canonname):
+                return False
+
+        return self.is_ip_allowed(ip, _local_addresses=_local_addresses)

cps/cw_advocate/api.py

@@ -0,0 +1,202 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+"""
+advocate.api
+~~~~~~~~~~~~
+
+This module implements the Requests API, largely a copy/paste from `requests`
+itself.
+
+:copyright: (c) 2015 by Jordan Milne.
+:license: Apache2, see LICENSE for more details.
+
+"""
+from collections import OrderedDict
+import hashlib
+import pickle
+
+from requests import Session as RequestsSession
+
+# import cw_advocate
+from .adapters import ValidatingHTTPAdapter
+from .exceptions import MountDisabledException
+
+
+class Session(RequestsSession):
+    """Convenience wrapper around `requests.Session` set up for `advocate`ing"""
+
+    __attrs__ = RequestsSession.__attrs__ + ["validator"]
+    DEFAULT_VALIDATOR = None
+    """
+    User-replaceable default validator to use for all Advocate sessions,
+    includes sessions created by advocate.get()
+    """
+
+    def __init__(self, *args, **kwargs):
+        self.validator = kwargs.pop("validator", None) or self.DEFAULT_VALIDATOR
+        adapter_kwargs = kwargs.pop("_adapter_kwargs", {})
+
+        # `Session.__init__()` calls `mount()` internally, so we need to allow
+        # it temporarily
+        self.__mount_allowed = True
+        RequestsSession.__init__(self, *args, **kwargs)
+
+        # Drop any existing adapters
+        self.adapters = OrderedDict()
+
+        self.mount("http://", ValidatingHTTPAdapter(validator=self.validator, **adapter_kwargs))
+        self.mount("https://", ValidatingHTTPAdapter(validator=self.validator, **adapter_kwargs))
+        self.__mount_allowed = False
+
+    def mount(self, *args, **kwargs):
+        """Wrapper around `mount()` to prevent a protection bypass"""
+        if self.__mount_allowed:
+            super().mount(*args, **kwargs)
+        else:
+            raise MountDisabledException(
+                "mount() is disabled to prevent protection bypasses"
+            )
+
+
+def session(*args, **kwargs):
+    return Session(*args, **kwargs)
+
+
+def request(method, url, **kwargs):
+    """Constructs and sends a :class:`Request <Request>`.
+
+    :param method: method for the new :class:`Request` object.
+    :param url: URL for the new :class:`Request` object.
+    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
+    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+    :param json: (optional) json data to send in the body of the :class:`Request`.
+    :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
+    :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
+    :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``) for multipart encoding upload.
+    :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
+    :param timeout: (optional) How long to wait for the server to send data
+        before giving up, as a float, or a (`connect timeout, read timeout
+        <user/advanced.html#timeouts>`_) tuple.
+    :type timeout: float or tuple
+    :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
+    :type allow_redirects: bool
+    :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
+    :param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
+    :param stream: (optional) if ``False``, the response content will be immediately downloaded.
+    :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
+    """
+
+    validator = kwargs.pop("validator", None)
+    with Session(validator=validator) as sess:
+        response = sess.request(method=method, url=url, **kwargs)
+    return response
+
+
+def get(url, **kwargs):
+    """Sends a GET request.
+
+    :param url: URL for the new :class:`Request` object.
+    :param **kwargs: Optional arguments that ``request`` takes.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
+    """
+
+    kwargs.setdefault('allow_redirects', True)
+    return request('get', url, **kwargs)
+
+
+class RequestsAPIWrapper:
+    """Provides a `requests.api`-like interface with a specific validator"""
+
+    # Due to how the classes are dynamically constructed pickling may not work
+    # correctly unless loaded within the same interpreter instance.
+    # Enable at your peril.
+    SUPPORT_WRAPPER_PICKLING = False
+
+    def __init__(self, validator):
+        # Do this here to avoid circular import issues
+        try:
+            from .futures import FuturesSession
+            have_requests_futures = True
+        except ImportError as e:
+            have_requests_futures = False
+
+        self.validator = validator
+        outer_self = self
+
+        class _WrappedSession(Session):
+            """An `advocate.Session` that uses the wrapper's blacklist
+
+            the wrapper is meant to be a transparent replacement for `requests`,
+            so people should be able to subclass `wrapper.Session` and still
+            get the desired validation behaviour
+            """
+            DEFAULT_VALIDATOR = outer_self.validator
+
+        self._make_wrapper_cls_global(_WrappedSession)
+
+        if have_requests_futures:
+
+            class _WrappedFuturesSession(FuturesSession):
+                """Like _WrappedSession, but for `FuturesSession`s"""
+                DEFAULT_VALIDATOR = outer_self.validator
+            self._make_wrapper_cls_global(_WrappedFuturesSession)
+
+            self.FuturesSession = _WrappedFuturesSession
+
+        self.request = self._default_arg_wrapper(request)
+        self.get = self._default_arg_wrapper(get)
+        self.Session = _WrappedSession
+
+    def __getattr__(self, item):
+        # This class is meant to mimic the requests base module, so if we don't
+        # have this attribute, it might be on the base module (like the Request
+        # class, etc.)
+        try:
+            return object.__getattribute__(self, item)
+        except AttributeError:
+            from . import cw_advocate
+            return getattr(cw_advocate, item)
+
+    def _default_arg_wrapper(self, fun):
+        def wrapped_func(*args, **kwargs):
+            kwargs.setdefault("validator", self.validator)
+            return fun(*args, **kwargs)
+        return wrapped_func
+
+    def _make_wrapper_cls_global(self, cls):
+        if not self.SUPPORT_WRAPPER_PICKLING:
+            return
+        # Gnarly, but necessary to give pickle a consistent module-level
+        # reference for each wrapper.
+        wrapper_hash = hashlib.sha256(pickle.dumps(self)).hexdigest()
+        cls.__name__ = "_".join((cls.__name__, wrapper_hash))
+        cls.__qualname__ = ".".join((__name__, cls.__name__))
+        if not globals().get(cls.__name__):
+            globals()[cls.__name__] = cls
+
+
+__all__ = (
+    "get",
+    "request",
+    "session",
+    "Session",
+    "RequestsAPIWrapper",
+)

cps/cw_advocate/connection.py

@@ -0,0 +1,201 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+import ipaddress
+import socket
+from socket import timeout as SocketTimeout
+
+from urllib3.connection import HTTPSConnection, HTTPConnection
+from urllib3.exceptions import ConnectTimeoutError
+from urllib3.util.connection import _set_socket_options
+from urllib3.util.connection import create_connection as old_create_connection
+
+from . import addrvalidator
+from .exceptions import UnacceptableAddressException
+
+
+def advocate_getaddrinfo(host, port, get_canonname=False):
+    addrinfo = socket.getaddrinfo(
+        host,
+        port,
+        0,
+        socket.SOCK_STREAM,
+        0,
+        # We need what the DNS client sees the hostname as, correctly handles
+        # IDNs and tricky things like `private.foocorp.org\x00.google.com`.
+        # All IDNs will be converted to punycode.
+        socket.AI_CANONNAME if get_canonname else 0,
+    )
+    return fix_addrinfo(addrinfo)
+
+
+def fix_addrinfo(records):
+    """
+    Propagate the canonname across records and parse IPs
+
+    I'm not sure if this is just the behaviour of `getaddrinfo` on Linux, but
+    it seems like only the first record in the set has the canonname field
+    populated.
+    """
+    def fix_record(record, canonname):
+        sa = record[4]
+        sa = (ipaddress.ip_address(sa[0]),) + sa[1:]
+        return record[0], record[1], record[2], canonname, sa
+
+    canonname = None
+    if records:
+        # Apparently the canonical name is only included in the first record?
+        # Add it to all of them.
+        assert(len(records[0]) == 5)
+        canonname = records[0][3]
+    return tuple(fix_record(x, canonname) for x in records)
+
+
+# Lifted from requests' urllib3, which in turn lifted it from `socket.py`. Oy!
+def validating_create_connection(address,
+                       timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
+                       source_address=None, socket_options=None,
+                       validator=None):
+    """Connect to *address* and return the socket object.
+
+    Convenience function.  Connect to *address* (a 2-tuple ``(host,
+    port)``) and return the socket object.  Passing the optional
+    *timeout* parameter will set the timeout on the socket instance
+    before attempting to connect.  If no *timeout* is supplied, the
+    global default timeout setting returned by :func:`getdefaulttimeout`
+    is used.  If *source_address* is set it must be a tuple of (host, port)
+    for the socket to bind as a source address before making the connection.
+    An host of '' or port 0 tells the OS to use the default.
+    """
+
+    host, port = address
+    # We can skip asking for the canon name if we're not doing hostname-based
+    # blacklisting.
+    need_canonname = False
+    if validator.hostname_blacklist:
+        need_canonname = True
+        # We check both the non-canonical and canonical hostnames so we can
+        # catch both of these:
+        # CNAME from nonblacklisted.com -> blacklisted.com
+        # CNAME from blacklisted.com -> nonblacklisted.com
+        if not validator.is_hostname_allowed(host):
+            raise UnacceptableAddressException(host)
+
+    err = None
+    addrinfo = advocate_getaddrinfo(host, port, get_canonname=need_canonname)
+    if addrinfo:
+        if validator.autodetect_local_addresses:
+            local_addresses = addrvalidator.determine_local_addresses()
+        else:
+            local_addresses = []
+        for res in addrinfo:
+            # Are we allowed to connect with this result?
+            if not validator.is_addrinfo_allowed(
+                res,
+                _local_addresses=local_addresses,
+            ):
+                continue
+            af, socktype, proto, canonname, sa = res
+            # Unparse the validated IP
+            sa = (sa[0].exploded,) + sa[1:]
+            sock = None
+            try:
+                sock = socket.socket(af, socktype, proto)
+
+                # If provided, set socket level options before connecting.
+                # This is the only addition urllib3 makes to this function.
+                _set_socket_options(sock, socket_options)
+
+                if timeout is not socket._GLOBAL_DEFAULT_TIMEOUT:
+                    sock.settimeout(timeout)
+                if source_address:
+                    sock.bind(source_address)
+                sock.connect(sa)
+                return sock
+
+            except socket.error as _:
+                err = _
+                if sock is not None:
+                    sock.close()
+                    sock = None
+
+        if err is None:
+            # If we got here, none of the results were acceptable
+            err = UnacceptableAddressException(address)
+    if err is not None:
+        raise err
+    else:
+        raise socket.error("getaddrinfo returns an empty list")
+
+
+# TODO: Is there a better way to add this to multiple classes with different
+# base classes? I tried a mixin, but it used the base method instead.
+def _validating_new_conn(self):
+    """ Establish a socket connection and set nodelay settings on it.
+
+    :return: New socket connection.
+    """
+    extra_kw = {}
+    if self.source_address:
+        extra_kw['source_address'] = self.source_address
+
+    if self.socket_options:
+        extra_kw['socket_options'] = self.socket_options
+
+    try:
+        # Hack around HTTPretty's patched sockets
+        # TODO: some better method of hacking around it that checks if we
+        # _would have_ connected to a private addr?
+        conn_func = validating_create_connection
+        if socket.getaddrinfo.__module__.startswith("httpretty"):
+            conn_func = old_create_connection
+        else:
+            extra_kw["validator"] = self._validator
+
+        conn = conn_func(
+            (self.host, self.port),
+            self.timeout,
+            **extra_kw
+        )
+
+    except SocketTimeout:
+        raise ConnectTimeoutError(
+            self, "Connection to %s timed out. (connect timeout=%s)" %
+            (self.host, self.timeout))
+
+    return conn
+
+
+# Don't silently break if the private API changes across urllib3 versions
+assert(hasattr(HTTPConnection, '_new_conn'))
+assert(hasattr(HTTPSConnection, '_new_conn'))
+
+
+class ValidatingHTTPConnection(HTTPConnection):
+    _new_conn = _validating_new_conn
+
+    def __init__(self, *args, **kwargs):
+        self._validator = kwargs.pop("validator")
+        HTTPConnection.__init__(self, *args, **kwargs)
+
+
+class ValidatingHTTPSConnection(HTTPSConnection):
+    _new_conn = _validating_new_conn
+
+    def __init__(self, *args, **kwargs):
+        self._validator = kwargs.pop("validator")
+        HTTPSConnection.__init__(self, *args, **kwargs)

cps/cw_advocate/connectionpool.py

@@ -0,0 +1,39 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+from urllib3 import HTTPConnectionPool, HTTPSConnectionPool
+
+from .connection import (
+    ValidatingHTTPConnection,
+    ValidatingHTTPSConnection,
+)
+
+# Don't silently break if the private API changes across urllib3 versions
+assert(hasattr(HTTPConnectionPool, 'ConnectionCls'))
+assert(hasattr(HTTPSConnectionPool, 'ConnectionCls'))
+assert(hasattr(HTTPConnectionPool, 'scheme'))
+assert(hasattr(HTTPSConnectionPool, 'scheme'))
+
+
+class ValidatingHTTPConnectionPool(HTTPConnectionPool):
+    scheme = 'http'
+    ConnectionCls = ValidatingHTTPConnection
+
+
+class ValidatingHTTPSConnectionPool(HTTPSConnectionPool):
+    scheme = 'https'
+    ConnectionCls = ValidatingHTTPSConnection

cps/cw_advocate/exceptions.py

@@ -0,0 +1,39 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+class AdvocateException(Exception):
+    pass
+
+
+class UnacceptableAddressException(AdvocateException):
+    pass
+
+
+class NameserverException(AdvocateException):
+    pass
+
+
+class MountDisabledException(AdvocateException):
+    pass
+
+
+class ProxyDisabledException(NotImplementedError, AdvocateException):
+    pass
+
+
+class ConfigException(AdvocateException):
+    pass

cps/cw_advocate/poolmanager.py

@@ -0,0 +1,61 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+import collections
+import functools
+
+from urllib3 import PoolManager
+from urllib3.poolmanager import _default_key_normalizer, PoolKey
+
+from .connectionpool import (
+    ValidatingHTTPSConnectionPool,
+    ValidatingHTTPConnectionPool,
+)
+
+pool_classes_by_scheme = {
+    "http": ValidatingHTTPConnectionPool,
+    "https": ValidatingHTTPSConnectionPool,
+}
+
+AdvocatePoolKey = collections.namedtuple('AdvocatePoolKey',
+                                         PoolKey._fields + ('key_validator',))
+
+
+def key_normalizer(key_class, request_context):
+    request_context = request_context.copy()
+    # TODO: add ability to serialize validator rules to dict,
+    # allowing pool to be shared between sessions with the same
+    # rules.
+    request_context["validator"] = id(request_context["validator"])
+    return _default_key_normalizer(key_class, request_context)
+
+
+key_fn_by_scheme = {
+    'http': functools.partial(key_normalizer, AdvocatePoolKey),
+    'https': functools.partial(key_normalizer, AdvocatePoolKey),
+}
+
+
+class ValidatingPoolManager(PoolManager):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+        # Make sure the API hasn't changed
+        assert (hasattr(self, 'pool_classes_by_scheme'))
+
+        self.pool_classes_by_scheme = pool_classes_by_scheme
+        self.key_fn_by_scheme = key_fn_by_scheme.copy()

cps/db.py

@@ -114,7 +114,7 @@ class Identifiers(Base):
         if format_type == 'amazon':
             return "Amazon"
         elif format_type.startswith("amazon_"):
-            return "Amazon.{0}".format(format_type[7:])
+            return "Amazon.{0}".format(format_type[7:].lower().replace("uk","co.uk"))
         elif format_type == "isbn":
             return "ISBN"
         elif format_type == "doi":
@@ -149,7 +149,7 @@ class Identifiers(Base):
         if format_type == "amazon" or format_type == "asin":
             return "https://amazon.com/dp/{0}".format(self.val)
         elif format_type.startswith('amazon_'):
-            return "https://amazon.{0}/dp/{1}".format(format_type[7:], self.val)
+            return "https://amazon.{0}/dp/{1}".format(format_type[7:].lower().replace("uk","co.uk"), self.val)
         elif format_type == "isbn":
             return "https://www.worldcat.org/isbn/{0}".format(self.val)
         elif format_type == "doi":

cps/dep_check.py

@@ -39,11 +39,27 @@ def load_dependencies(optional=False):
             with open(req_path, 'r') as f:
                 for line in f:
                     if not line.startswith('#') and not line == '\n' and not line.startswith('git'):
-                        res = re.match(r'(.*?)([<=>\s]+)([\d\.]+),?\s?([<=>\s]+)?([\d\.]+)?', line.strip())
+                        res = re.match(r'(.*?)([<=>\s]+)([\d\.]+),?\s?([<=>\s]+)?([\d\.]+)?(?:\s?;\s?'
+                                       r'(?:(python_version)\s?([<=>]+)\s?\'([\d\.]+)\'|'
+                                       r'(sys_platform)\s?([\!=]+)\s?\'([\w]+)\'))?', line.strip())
                         try:
                             if getattr(sys, 'frozen', False):
                                 dep_version = exe_deps[res.group(1).lower().replace('_', '-')]
                             else:
+                                if res.group(7) and res.group(8):
+                                    val = res.group(8).split(".")
+                                    if not eval(str(sys.version_info[0]) + "." + "{:02d}".format(sys.version_info[1]) +
+                                                res.group(7) + val[0] + "." + "{:02d}".format(int(val[1]))):
+                                        continue
+                                elif res.group(10) and res.group(11):
+                                    # only installed if platform is eqal, don't check if platform is not equal
+                                    if res.group(10) == "==":
+                                        if sys.platform != res.group(11):
+                                            continue
+                                    # installed if platform is not eqal, don't check if platform is equal
+                                    elif res.group(10) == "!=":
+                                        if sys.platform == res.group(11):
+                                            continue
                                 if importlib:
                                     dep_version = version(res.group(1))
                                 else:
@@ -63,7 +79,7 @@ def dependency_check(optional=False):
     deps = load_dependencies(optional)
     for dep in deps:
         try:
-            dep_version_int = [int(x) if x.isnumeric() else 0 for x in dep[0].split('.')]
+            dep_version_int = [int(x) if x.isnumeric() else 0 for x in dep[0].split('.')[:3]]
             low_check = [int(x) for x in dep[3].split('.')]
             high_check = [int(x) for x in dep[5].split('.')]
         except AttributeError:

cps/editbooks.py

@@ -501,6 +501,7 @@ def do_edit_book(book_id, upload_formats=None):
                     modify_date = True
                     helper.replace_cover_thumbnail_cache(book.id)
                 else:
+                    edit_error = True
                     flash(error, category="error")
 
         # Add default series_index to book

cps/helper.py

@@ -43,10 +43,10 @@ from markupsafe import escape
 from urllib.parse import quote
 
 try:
-    import advocate
-    from advocate.exceptions import UnacceptableAddressException
+    from . import cw_advocate
+    from .cw_advocate.exceptions import UnacceptableAddressException
     use_advocate = True
-except ImportError:
+except ImportError as e:
     use_advocate = False
     advocate = requests
     UnacceptableAddressException = MissingSchema = BaseException
@@ -418,8 +418,6 @@ def rename_author_path(first_author, old_author_dir, renamed_author, calibre_pat
     # Create new_author_dir from parameter or from database
     # Create new title_dir from database and add id
     new_authordir = get_valid_filename(first_author, chars=96)
-    # new_author = calibre_db.session.query(db.Authors).filter(db.Authors.name == renamed_author).first()
-    # old_author_dir = get_valid_filename(old_author_name, chars=96)
     new_author_rename_dir = get_valid_filename(renamed_author, chars=96)
     if gdrive:
         g_file = gd.getFileFromEbooksFolder(None, old_author_dir)
@@ -468,7 +466,6 @@ def update_dir_structure_file(book_id, calibre_path, original_filepath, new_auth
                                      db_filename,
                                      original_filepath,
                                      path)
-        # old_path = os.path.join(calibre_path, author_dir, new_title_dir).replace('\\', '/')
         new_path = os.path.join(calibre_path, new_author_dir, new_title_dir).replace('\\', '/')
         all_new_name = get_valid_filename(local_book.title, chars=42) + ' - ' \
                        + get_valid_filename(new_author, chars=42)
@@ -477,8 +474,6 @@ def update_dir_structure_file(book_id, calibre_path, original_filepath, new_auth
 
         if error:
             return error
-
-    # Rename all files from old names to new names
     return False
 
 
@@ -490,7 +485,7 @@ def upload_new_file_gdrive(book_id, first_author, title, title_dir, original_fil
                                title_dir + " (" + str(book_id) + ")")
     book.path = gdrive_path.replace("\\", "/")
     gd.uploadFileToEbooksFolder(os.path.join(gdrive_path, file_name).replace("\\", "/"), original_filepath)
-    return False # rename_files_on_change(first_author, renamed_author, local_book=book, gdrive=True)
+    return False
 
 
 def update_dir_structure_gdrive(book_id, first_author):
@@ -523,19 +518,21 @@ def update_dir_structure_gdrive(book_id, first_author):
         all_new_name = get_valid_filename(book.title, chars=42) + ' - ' \
                        + get_valid_filename(new_authordir, chars=42)
         rename_all_files_on_change(book, book.path, book.path, all_new_name, gdrive=True)  # todo: Move filenames on gdrive
-    # change location in database to new author/title path
-    # book.path = os.path.join(authordir, new_titledir).replace('\\', '/')
     return False
 
 
 def move_files_on_change(calibre_path, new_author_dir, new_titledir, localbook, db_filename, original_filepath, path):
     new_path = os.path.join(calibre_path, new_author_dir, new_titledir)
-    # new_name = get_valid_filename(localbook.title, chars=96) + ' - ' + new_author_dir
     try:
         if original_filepath:
             if not os.path.isdir(new_path):
                 os.makedirs(new_path)
-            shutil.move(original_filepath, os.path.join(new_path, db_filename))
+            try:
+                shutil.move(original_filepath, os.path.join(new_path, db_filename))
+            except OSError:
+                log.error("Rename title from {} to {} failed with error, trying to "
+                          "move without metadata".format(path, new_path))
+                shutil.move(original_filepath, os.path.join(new_path, db_filename), copy_function=shutil.copy)
             log.debug("Moving title: %s to %s", original_filepath, new_path)
         else:
             # Check new path is not valid path
@@ -844,7 +841,7 @@ def save_cover_from_url(url, book_path):
         if cli_param.allow_localhost:
             img = requests.get(url, timeout=(10, 200), allow_redirects=False)  # ToDo: Error Handling
         elif use_advocate:
-            img = advocate.get(url, timeout=(10, 200), allow_redirects=False)      # ToDo: Error Handling
+            img = cw_advocate.get(url, timeout=(10, 200), allow_redirects=False)      # ToDo: Error Handling
         else:
             log.error("python module advocate is not installed but is needed")
             return False, _("Python module 'advocate' is not installed but is needed for cover uploads")

cps/isoLanguages.py

@@ -25,20 +25,7 @@ log = logger.create()
 
 
 try:
-    from iso639 import languages
-    # iso_version = importlib.metadata.version("iso639")
-    get = languages.get
-    try:
-        if sys.version_info >= (3, 12):
-            import pkg_resources
-    except ImportError:
-        print("Python 3.12 isn't compatible with iso-639. Please install pycountry.")
-except ImportError as ex:
     from pycountry import languages as pyc_languages
-    #try:
-    #    iso_version = importlib.metadata.version("pycountry") + ' (PyCountry)'
-    #except (ImportError, Exception):
-    #    iso_version = "?" + ' (PyCountry)'
 
     def _copy_fields(l):
         l.part1 = getattr(l, 'alpha_2', None)
@@ -52,6 +39,11 @@ except ImportError as ex:
             return _copy_fields(pyc_languages.get(alpha_2=part1))
         if name is not None:
             return _copy_fields(pyc_languages.get(name=name))
+except ImportError as ex:
+    if sys.version_info >= (3, 12):
+        print("Python 3.12 isn't compatible with iso-639. Please install pycountry.")
+    from iso639 import languages
+    get = languages.get
 
 
 def get_language_names(locale):
@@ -109,6 +101,6 @@ def get_lang3(lang):
             ret_value = lang
         else:
             ret_value = ""
-    except KeyError:
+    except (KeyError, AttributeError):
         ret_value = lang
     return ret_value

cps/iso_language_names.py

@@ -8138,6 +8138,384 @@ LANGUAGE_NAMES = {
         "zul": "Zulu",
         "zun": "Zuni"
     },
+    "sl": {
+        "abk": "abhazijski",
+        "ace": "achinese",
+        "ach": "Acoli",
+        "ada": "Adangme",
+        "ady": "Adyghe",
+        "aar": "afarski",
+        "afh": "Afrihili",
+        "afr": "afrikanski",
+        "ain": "Ainu (Japan)",
+        "aka": "Akan",
+        "akk": "akadski",
+        "sqi": "albanščina",
+        "ale": "aleutski",
+        "amh": "amharski",
+        "anp": "Angika",
+        "ara": "arabski",
+        "arg": "aragonski",
+        "arp": "Arapaho",
+        "arw": "araukanski",
+        "hye": "armenščina",
+        "asm": "asamski",
+        "ast": "Asturian",
+        "ava": "avarski",
+        "ave": "avestijski jeziki",
+        "awa": "Awadhi",
+        "aym": "Aymara",
+        "aze": "azerbajdžanski",
+        "ban": "balijščina",
+        "bal": "belučijski",
+        "bam": "bambarski",
+        "bas": "Basa (Cameroon)",
+        "bak": "baškirski",
+        "eus": "baskovščina",
+        "bej": "Beja",
+        "bel": "beloruščina",
+        "bem": "Bemba (Zambia)",
+        "ben": "bengalščina",
+        "bit": "Berinomo",
+        "bho": "Bhojpuri",
+        "bik": "bikolščina",
+        "byn": "Bilin",
+        "bin": "Bini",
+        "bis": "bislama",
+        "zbl": "Blissymbols",
+        "bos": "bošnjaščina",
+        "bra": "Braj",
+        "bre": "bretonščina",
+        "bug": "buginščina",
+        "bul": "bolgarščina",
+        "bua": "burjatščina",
+        "mya": "burmanščina",
+        "cad": "kadajščina?",
+        "cat": "katalonščina",
+        "ceb": "cebuanščina",
+        "chg": "Chagatai",
+        "cha": "čamorščina",
+        "che": "čečenščina",
+        "chr": "čerokeščina",
+        "chy": "čejenščina",
+        "chb": "čibčevščina",
+        "zho": "kitajščina",
+        "chn": "Chinook jargon",
+        "chp": "čipevščina",
+        "cho": "Choctaw",
+        "cht": "Cholón",
+        "chk": "Chuukese",
+        "chv": "čuvaščina",
+        "cop": "koptščina",
+        "cor": "kornijščina",
+        "cos": "korzijščina",
+        "cre": "krijščina",
+        "mus": "Creek",
+        "hrv": "hrvaščina",
+        "ces": "češčina",
+        "dak": "Dakota",
+        "dan": "danski",
+        "dar": "Dargwa",
+        "del": "Delaware",
+        "div": "Dhivehi",
+        "din": "Dinka",
+        "doi": "Dogri (macrolanguage)",
+        "dgr": "Dogrib",
+        "dua": "Duala",
+        "nld": "nizozemščina",
+        "dse": "Dutch Sign Language",
+        "dyu": "Dyula",
+        "dzo": "dzongkha",
+        "efi": "Efik",
+        "egy": "egipčanski",
+        "eka": "Ekajuk",
+        "elx": "elamščina",
+        "eng": "angleščina",
+        "enu": "Enu",
+        "myv": "Erzya",
+        "epo": "esperanto",
+        "est": "estonščina",
+        "ewe": "evenščina",
+        "ewo": "Ewondo",
+        "fan": "Fang (Equatorial Guinea)",
+        "fat": "Fanti",
+        "fao": "ferščina",
+        "fij": "fidžijščina",
+        "fil": "Filipino",
+        "fin": "finščina",
+        "fon": "Fon",
+        "fra": "francoščina",
+        "fur": "furlanščina",
+        "ful": "fulščina",
+        "gaa": "Ga",
+        "glg": "Galician",
+        "lug": "Ganda",
+        "gay": "gajščina?",
+        "gba": "Gbaya (Central African Republic)",
+        "hmj": "Ge",
+        "gez": "etiopščina?",
+        "kat": "gruzinščina",
+        "deu": "nemški",
+        "gil": "gilbertščina",
+        "gon": "Gondi",
+        "gor": "Gorontalo",
+        "got": "gotščina",
+        "grb": "Grebo",
+        "grn": "gvaranijščina",
+        "guj": "gudžaratščina",
+        "gwi": "Gwichʼin",
+        "hai": "haidščina",
+        "hau": "havščina",
+        "haw": "havajščina",
+        "heb": "hebrejščina",
+        "her": "Herero",
+        "hil": "hilingajnonščina",
+        "hin": "hindijščina",
+        "hmo": "hiri motu",
+        "hit": "hetitščina",
+        "hmn": "hmonščina; miaojščina",
+        "hun": "madžarščina",
+        "hup": "hupščina",
+        "iba": "ibanščina",
+        "isl": "islandščina",
+        "ido": "Ido",
+        "ibo": "Igbo",
+        "ilo": "Iloko",
+        "ind": "indonezijščina",
+        "inh": "inguščina",
+        "ina": "interlingva",
+        "ile": "Interlingue",
+        "iku": "inuktituščina",
+        "ipk": "Inupiaq",
+        "gle": "irščina",
+        "ita": "italijanščina",
+        "jpn": "japonščina",
+        "jav": "javanščina",
+        "jrb": "Judeo-Arabic",
+        "jpr": "Judeo-Persian",
+        "kbd": "kabardinščina",
+        "kab": "Kabyle",
+        "kac": "Kachin",
+        "kal": "Kalaallisut",
+        "xal": "Kalmyk",
+        "kam": "Kamba (Kenya)",
+        "kan": "kanareščina",
+        "kau": "Kanuri",
+        "kaa": "Kara-Kalpak",
+        "krc": "Karachay-Balkar",
+        "krl": "Karelian",
+        "kas": "kašmirščina",
+        "csb": "Kashubian",
+        "kaw": "kavi",
+        "kaz": "kazaščina",
+        "kha": "Khasi",
+        "kho": "Khotanese",
+        "kik": "kikujščina",
+        "kmb": "Kimbundu",
+        "kin": "Kinyarwanda",
+        "kir": "kirgiščina",
+        "tlh": "Klingon",
+        "kom": "komijščina",
+        "kon": "Kongo",
+        "kok": "Konkani (macrolanguage)",
+        "kor": "korejščina",
+        "kos": "Kosraean",
+        "kpe": "Kpelle",
+        "kua": "Kuanyama",
+        "kum": "kumiščina",
+        "kur": "kurdščina",
+        "kru": "Kurukh",
+        "kut": "kutenajščina",
+        "lad": "ladinščina",
+        "lah": "Lahnda",
+        "lam": "Lamba",
+        "lao": "laoščina",
+        "lat": "latinščina",
+        "lav": "latvijščina",
+        "lez": "lezginščina",
+        "lim": "Limburgan",
+        "lin": "lingala",
+        "lit": "litvanščina",
+        "jbo": "Lojban",
+        "loz": "Lozi",
+        "lub": "Luba-Katanga",
+        "lua": "lubalulujščina",
+        "lui": "Luiseno",
+        "smj": "Lule Sami",
+        "lun": "Lunda",
+        "luo": "Luo (Kenya and Tanzania)",
+        "lus": "Lushai",
+        "ltz": "Luxembourgish",
+        "mkd": "makedonščina",
+        "mad": "madurščina",
+        "mag": "Magahi",
+        "mai": "Maithili",
+        "mak": "makasarščina",
+        "mlg": "malgaščina",
+        "msa": "Malay (macrolanguage)",
+        "mal": "malajalščina",
+        "mlt": "malteščina",
+        "mnc": "Manchu",
+        "mdr": "Mandar",
+        "man": "Mandingo",
+        "mni": "manipurščina",
+        "glv": "manska gelščina",
+        "mri": "maorščina",
+        "arn": "Mapudungun",
+        "mar": "maratščina",
+        "chm": "Mari (Russia)",
+        "mah": "Marshallese",
+        "mwr": "Marwari",
+        "mas": "masajščina",
+        "men": "Mende (Sierra Leone)",
+        "mic": "Mi'kmaq",
+        "min": "Minangkabau",
+        "mwl": "Mirandese",
+        "moh": "mohoščina",
+        "mdf": "Moksha",
+        "lol": "Mongo",
+        "mon": "mongolščina",
+        "mos": "mosanščina",
+        "mul": "Več jezikov",
+        "nqo": "N'Ko",
+        "nau": "Nauru",
+        "nav": "navaščina",
+        "ndo": "Ndonga",
+        "nap": "napolitanščina",
+        "nia": "niaščina",
+        "niu": "niuejščina",
+        "zxx": "No linguistic content",
+        "nog": "Nogai",
+        "nor": "norveščina",
+        "nob": "Norwegian Bokmål",
+        "nno": "norveščina; nynorsk",
+        "nym": "Nyamwezi",
+        "nya": "Nyanja",
+        "nyn": "Nyankole",
+        "nyo": "Nyoro",
+        "nzi": "Nzima",
+        "oci": "Occitan (post 1500)",
+        "oji": "Ojibwa",
+        "orm": "Oromo",
+        "osa": "Osage",
+        "oss": "Ossetian",
+        "pal": "Pahlavi",
+        "pau": "palavanščina",
+        "pli": "Pali",
+        "pam": "Pampanga",
+        "pag": "pangasinanščina",
+        "pan": "Panjabi",
+        "pap": "papiamentu",
+        "fas": "perzijščina",
+        "phn": "feničanščina",
+        "pon": "Pohnpeian",
+        "pol": "poljščina",
+        "por": "portugalđščina",
+        "pus": "paštu",
+        "que": "Quechua",
+        "raj": "radžastanščina",
+        "rap": "rapanujščina",
+        "ron": "romunščina",
+        "roh": "Romansh",
+        "rom": "romščina",
+        "run": "rundščina",
+        "rus": "ruščina",
+        "smo": "samoanščina",
+        "sad": "Sandawe",
+        "sag": "Sango",
+        "san": "sanskrt",
+        "sat": "santalščina",
+        "srd": "sardinščina",
+        "sas": "Sasak",
+        "sco": "škotščina",
+        "sel": "selkupščina",
+        "srp": "srbščina",
+        "srr": "Serer",
+        "shn": "šanščina",
+        "sna": "šonščina",
+        "scn": "sicilijanščina",
+        "sid": "Sidamo",
+        "bla": "Siksika",
+        "snd": "sindščina",
+        "sin": "Sinhala",
+        "den": "Slave (Athapascan)",
+        "slk": "slovaščina",
+        "slv": "slovenščina",
+        "sog": "Sogdian",
+        "som": "Somali",
+        "snk": "Soninke",
+        "spa": "španščina",
+        "srn": "Sranan Tongo",
+        "suk": "Sukuma",
+        "sux": "sumerščina",
+        "sun": "sundščina",
+        "sus": "susuamijščina?",
+        "swa": "Swahili (macrolanguage)",
+        "ssw": "svazijščina?",
+        "swe": "švedščina",
+        "syr": "sirščina",
+        "tgl": "tagaloščina",
+        "tah": "tahitijščina",
+        "tgk": "tadžiščina",
+        "tmh": "Tamashek",
+        "tam": "tamilščina",
+        "tat": "tatarščina",
+        "tel": "Telugu",
+        "ter": "Tereno",
+        "tet": "Tetum",
+        "tha": "tajščina",
+        "bod": "tibetanščina",
+        "tig": "Tigre",
+        "tir": "Tigrinya",
+        "tem": "Timne",
+        "tiv": "Tiv",
+        "tli": "Tlingit",
+        "tpi": "tok pisin",
+        "tkl": "Tokelau",
+        "tog": "Tonga (Nyasa)",
+        "ton": "tonganščina",
+        "tsi": "tsimšijščina",
+        "tso": "Tsonga",
+        "tsn": "Tswana",
+        "tum": "Tumbuka",
+        "tur": "turščina",
+        "tuk": "turkmenščina",
+        "tvl": "tuvalujščina",
+        "tyv": "Tuvinian",
+        "twi": "Twi",
+        "udm": "Udmurt",
+        "uga": "ugaritščina",
+        "uig": "ujgurščina",
+        "ukr": "ukrajinščina",
+        "umb": "Umbundu",
+        "mis": "Uncoded languages",
+        "und": "nedoločen",
+        "urd": "urdujščina",
+        "uzb": "uzbeščina",
+        "vai": "vajščina",
+        "ven": "Venda",
+        "vie": "vietnamščina",
+        "vol": "Volapük",
+        "vot": "votjaščina",
+        "wln": "valonščina",
+        "war": "Waray (Philippines)",
+        "was": "Washo",
+        "cym": "valižanščina",
+        "wal": "Wolaytta",
+        "wol": "Wolof",
+        "xho": "koščina",
+        "sah": "jakutščina",
+        "yao": "jaojščina",
+        "yap": "Yapese",
+        "yid": "jidiš",
+        "yor": "jorubščina",
+        "zap": "Zapotec",
+        "zza": "Zaza",
+        "zen": "Zenaga",
+        "zha": "Zhuang",
+        "zul": "zulujščina",
+        "zun": "Zuni"
+    },
     "sv": {
         "aar": "Afar",
         "abk": "Abchaziska",

cps/jinjia.py

@@ -43,8 +43,6 @@ def url_for_other_page(page):
     args = request.view_args.copy()
     args['page'] = page
     for get, val in request.args.items():
-        if get == "page":
-            continue
         args[get] = val
     return url_for(request.endpoint, **args)
 
@@ -113,14 +111,15 @@ def yesno(value, yes, no):
 
 @jinjia.app_template_filter('formatfloat')
 def formatfloat(value, decimals=1):
-    value = 0 if not value else value
+    if not value:
+        return value
     formated_value = ('{0:.' + str(decimals) + 'f}').format(value)
     if formated_value.endswith('.' + "0" * decimals):
         formated_value = formated_value.rstrip('0').rstrip('.')
     return formated_value
 
 
-@jinjia.app_template_filter('formatseriesindex')
+'''@jinjia.app_template_filter('formatseriesindex')
 def formatseriesindex_filter(series_index):
     if series_index:
         try:
@@ -131,6 +130,7 @@ def formatseriesindex_filter(series_index):
         except (ValueError, TypeError):
             return series_index
     return 0
+'''
 
 
 @jinjia.app_template_filter('escapedlink')

cps/kobo.py

@@ -423,7 +423,7 @@ def get_series(book):
 
 
 def get_seriesindex(book):
-    return book.series_index or 1
+    return book.series_index if isinstance(book.series_index, float) else 1
 
 
 def get_language(book):
@@ -486,14 +486,16 @@ def get_metadata(book):
 
     if get_series(book):
         name = get_series(book)
-        metadata["Series"] = {
-            "Name": get_series(book),
-            "Number": get_seriesindex(book),        # ToDo Check int() ?
-            "NumberFloat": float(get_seriesindex(book)),
-            # Get a deterministic id based on the series name.
-            "Id": str(uuid.uuid3(uuid.NAMESPACE_DNS, name)),
-        }
-
+        try:
+            metadata["Series"] = {
+                "Name": get_series(book),
+                "Number": get_seriesindex(book),        # ToDo Check int() ?
+                "NumberFloat": float(get_seriesindex(book)),
+                # Get a deterministic id based on the series name.
+                "Id": str(uuid.uuid3(uuid.NAMESPACE_DNS, name)),
+            }
+        except Exception as e:
+            print(e)
     return metadata
 
 

cps/main.py

@@ -31,7 +31,6 @@ def main():
     app = create_app()
 
     from .web import web
-    from .basic import basic
     from .opds import opds
     from .admin import admi
     from .gdrive import gdrive
@@ -65,7 +64,6 @@ def main():
     app.register_blueprint(search)
     app.register_blueprint(tasks)
     app.register_blueprint(web)
-    app.register_blueprint(basic)
     app.register_blueprint(opds)
     limiter.limit("3/minute", key_func=request_username)(opds)
     app.register_blueprint(jinjia)

cps/metadata_provider/amazon.py

@@ -38,14 +38,16 @@ class Amazon(Metadata):
     __name__ = "Amazon"
     __id__ = "amazon"
     headers = {'upgrade-insecure-requests': '1',
-               'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36',
-               'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
-               'sec-gpc': '1',
-               'sec-fetch-site': 'none',
-               'sec-fetch-mode': 'navigate',
-               'sec-fetch-user': '?1',
-               'sec-fetch-dest': 'document',
-               'accept-encoding': 'gzip, deflate, br',
+               'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0',
+               'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8',
+               'Sec-Fetch-Site': 'same-origin',
+               'Sec-Fetch-Mode': 'navigate',
+               'Sec-Fetch-User': '?1',
+               'Sec-Fetch-Dest': 'document',
+               'Upgrade-Insecure-Requests': '1',
+               'Alt-Used' : 'www.amazon.com',
+               'Priority' : 'u=0, i',
+               'accept-encoding': 'gzip, deflate, br, zstd',
                'accept-language': 'en-US,en;q=0.9'}
     session = requests.Session()
     session.headers=headers
@@ -62,7 +64,7 @@ class Amazon(Metadata):
                     log.warning(ex)
                     return []
                 long_soup = BS(r.text, "lxml")  #~4sec :/
-                soup2 = long_soup.find("div", attrs={"cel_widget_id": "dpx-books-ppd_csm_instrumentation_wrapper"})
+                soup2 = long_soup.find("div", attrs={"cel_widget_id": "dpx-ppd_csm_instrumentation_wrapper"})
                 if soup2 is None:
                     return []
                 try:
@@ -106,7 +108,7 @@ class Amazon(Metadata):
                     except (AttributeError, ValueError):
                         match.rating = 0
                     try:
-                        match.cover = soup2.find("img", attrs={"class": "a-dynamic-image frontImage"})["src"]
+                        match.cover = soup2.find("img", attrs={"class": "a-dynamic-image"})["src"]
                     except (AttributeError, TypeError):
                         match.cover = ""
                     return match, index
@@ -132,7 +134,7 @@ class Amazon(Metadata):
             links_list = [next(filter(lambda i: "digital-text" in i["href"], x.findAll("a")))["href"] for x in
                           soup.findAll("div", attrs={"data-component-type": "s-search-result"})]
             with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
-                fut = {executor.submit(inner, link, index) for index, link in enumerate(links_list[:5])}
+                fut = {executor.submit(inner, link, index) for index, link in enumerate(links_list[:3])}
                 val = list(map(lambda x : x.result(), concurrent.futures.as_completed(fut)))
         result = list(filter(lambda x: x, val))
         return [x[0] for x in sorted(result, key=itemgetter(1))] #sort by amazons listing order for best relevance

cps/opds.py

@@ -424,11 +424,8 @@ def feed_shelf(book_id):
 @requires_basic_auth_if_no_ano
 def opds_download_link(book_id, book_format):
     if not auth.current_user().role_download():
-        return abort(403)
-    if "Kobo" in request.headers.get('User-Agent'):
-        client = "kobo"
-    else:
-        client = ""
+        return abort(401)
+    client = "kobo" if "Kobo" in request.headers.get('User-Agent') else ""
     return get_download_link(book_id, book_format.lower(), client)
 
 

cps/search.py

@@ -265,7 +265,7 @@ def render_adv_search_results(term, offset=None, order=None, limit=None):
     pub_end = term.get("publishend")
     rating_low = term.get("ratinghigh")
     rating_high = term.get("ratinglow")
-    description = term.get("comment")
+    description = term.get("comments")
     read_status = term.get("read_status")
     if author_name:
         author_name = strip_whitespaces(author_name).lower().replace(',', '|')

cps/search_metadata.py

@@ -33,7 +33,6 @@ from cps.services.Metadata import Metadata
 from . import constants, logger, ub, web_server
 from .usermanagement import user_login_required
 
-# current_milli_time = lambda: int(round(time() * 1000))
 
 meta = Blueprint("metadata", __name__)
 
@@ -130,7 +129,7 @@ def metadata_search():
     locale = get_locale()
     if query:
         static_cover = url_for("static", filename="generic_cover.jpg")
-        # start = current_milli_time()
+        # ret = cl[0].search(query, static_cover, locale)
         with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
             meta = {
                 executor.submit(c.search, query, static_cover, locale): c
@@ -139,5 +138,4 @@ def metadata_search():
             }
             for future in concurrent.futures.as_completed(meta):
                 data.extend([asdict(x) for x in future.result() if x])
-    # log.info({'Time elapsed {}'.format(current_milli_time()-start)})
     return Response(json.dumps(data), mimetype="application/json")

cps/services/background_scheduler.py

@@ -42,6 +42,7 @@ class BackgroundScheduler:
         if cls._instance is None:
             cls._instance = super(BackgroundScheduler, cls).__new__(cls)
             cls.log = logger.create()
+            logger.logging.getLogger('tzlocal').setLevel(logger.logging.WARNING)
             cls.scheduler = BScheduler()
             cls.scheduler.start()
 

cps/static/css/caliBlur.css

@@ -3268,6 +3268,10 @@ div.btn-group[role=group][aria-label="Download, send to Kindle, reading"] > div.
     left: auto !important
 }
 
+ul.dropdown-menu.offscreen {
+    margin: 2px -160px 0
+}
+
 div.btn-group[role=group][aria-label="Download, send to Kindle, reading"] .dropdown-menu.offscreen {
     position: fixed;
     top: 120px;
@@ -4333,6 +4337,7 @@ body.advanced_search > div.container-fluid > div.row-fluid > div.col-sm-10 > div
 
 .navbar-right > li > ul.dropdown-menu.offscreen {
     right: -10px
+
 }
 
 .login .plexBack, body.login > div.container-fluid > div.row-fluid > div.col-sm-2, body.login > div.navbar.navbar-default.navbar-static-top > div > form {
@@ -7951,3 +7956,5 @@ div.comments[data-readmore] {
     transition: height 300ms;
     overflow: hidden
 }
+
+.dropdown-menu > .offscreen

cps/static/css/epub_themes.css

@@ -15,5 +15,5 @@
 
 .blackTheme {
 	background: #000;
-	color: #fff
+	color: #fff;
 }

cps/static/css/main.css

@@ -77,7 +77,6 @@ body {
 }
 
 #panels a {
-  visibility: hidden;
   width: 18px;
   height: 20px;
   overflow: hidden;

cps/static/js/edit_books.js

@@ -3,7 +3,7 @@
  */
 /* global Bloodhound, language, Modernizr, tinymce, getPath */
 
-if ($("#comments").length) {
+if ($("#comments").length && typeof tinymce !== "undefined") {
     tinymce.init({
         selector: "#comments",
         plugins: 'code',

cps/static/js/libs/bootstrap-datepicker/locales/bootstrap-datepicker.sl.min.js

@@ -0,0 +1 @@
+!function(a){a.fn.datepicker.dates.sl={days:["Nedelja","Ponedeljek","Torek","Sreda","Četrtek","Petek","Sobota"],daysShort:["Ned","Pon","Tor","Sre","Čet","Pet","Sob"],daysMin:["Ne","Po","To","Sr","Če","Pe","So"],months:["Januar","Februar","Marec","April","Maj","Junij","Julij","Avgust","September","Oktober","November","December"],monthsShort:["Jan","Feb","Mar","Apr","Maj","Jun","Jul","Avg","Sep","Okt","Nov","Dec"],today:"Danes",weekStart:1}}(jQuery);

cps/static/js/table.js

@@ -685,6 +685,17 @@ function ratingFormatter(value, row) {
     return (value/2);
 }
 
+function seriesIndexFormatter(value, row) {
+    if (!value) {
+        return value;
+    }
+    formated_value = Number(value).toFixed(2);
+    if (formated_value.endsWith(".00")) {
+        formated_value = parseInt(formated_value).toString();
+    }
+    return formated_value;
+}
+
 
 /* Do some hiding disabling after user list is loaded */
 function loadSuccess() {
@@ -849,6 +860,7 @@ function BookCheckboxChange(checkbox, userId, field) {
         },
         success: handleListServerResponse
     });
+    console.log("test");
 }
 
 

cps/string_helper.py

@@ -19,5 +19,5 @@ import re
 
 
 def strip_whitespaces(text):
-    return re.sub("(^[\s\u200B-\u200D\ufeff]+)|([\s\u200B-\u200D\ufeff]+$)","", text)
+    return re.sub(r"(^[\s\u200B-\u200D\ufeff]+)|([\s\u200B-\u200D\ufeff]+$)","", text)
 

cps/tasks/convert.py

@@ -1,340 +1,351 @@
-# -*- coding: utf-8 -*-
-
-#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
-#    Copyright (C) 2020 pwr
-#
-#  This program is free software: you can redistribute it and/or modify
-#  it under the terms of the GNU General Public License as published by
-#  the Free Software Foundation, either version 3 of the License, or
-#  (at your option) any later version.
-#
-#  This program is distributed in the hope that it will be useful,
-#  but WITHOUT ANY WARRANTY; without even the implied warranty of
-#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#  GNU General Public License for more details.
-#
-#  You should have received a copy of the GNU General Public License
-#  along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-import os
-import re
-from glob import glob
-from shutil import copyfile, copyfileobj
-from markupsafe import escape
-from time import time
-from uuid import uuid4
-
-from sqlalchemy.exc import SQLAlchemyError
-from flask_babel import lazy_gettext as N_
-
-from cps.services.worker import CalibreTask
-from cps import db
-from cps import logger, config
-from cps.subproc_wrapper import process_open
-from flask_babel import gettext as _
-from cps.kobo_sync_status import remove_synced_book
-from cps.ub import init_db_thread
-from cps.file_helper import get_temp_dir
-
-from cps.tasks.mail import TaskEmail
-from cps import gdriveutils, helper
-from cps.constants import SUPPORTED_CALIBRE_BINARIES
-from cps.string_helper import strip_whitespaces
-
-log = logger.create()
-
-current_milli_time = lambda: int(round(time() * 1000))
-
-
-class TaskConvert(CalibreTask):
-    def __init__(self, file_path, book_id, task_message, settings, ereader_mail, user=None):
-        super(TaskConvert, self).__init__(task_message)
-        self.worker_thread = None
-        self.file_path = file_path
-        self.book_id = book_id
-        self.title = ""
-        self.settings = settings
-        self.ereader_mail = ereader_mail
-        self.user = user
-
-        self.results = dict()
-
-    def run(self, worker_thread):
-        self.worker_thread = worker_thread
-        if config.config_use_google_drive:
-            worker_db = db.CalibreDB(expire_on_commit=False, init=True)
-            cur_book = worker_db.get_book(self.book_id)
-            self.title = cur_book.title
-            data = worker_db.get_book_format(self.book_id, self.settings['old_book_format'])
-            df = gdriveutils.getFileFromEbooksFolder(cur_book.path,
-                                                     data.name + "." + self.settings['old_book_format'].lower())
-            df_cover = gdriveutils.getFileFromEbooksFolder(cur_book.path, "cover.jpg")
-            if df:
-                datafile_cover = None
-                datafile = os.path.join(config.get_book_path(),
-                                        cur_book.path,
-                                        data.name + "." + self.settings['old_book_format'].lower())
-                if df_cover:
-                    datafile_cover = os.path.join(config.get_book_path(),
-                                                  cur_book.path, "cover.jpg")
-                if not os.path.exists(os.path.join(config.get_book_path(), cur_book.path)):
-                    os.makedirs(os.path.join(config.get_book_path(), cur_book.path))
-                df.GetContentFile(datafile)
-                if df_cover:
-                    df_cover.GetContentFile(datafile_cover)
-                worker_db.session.close()
-            else:
-                # ToDo Include cover in error handling
-                error_message = _("%(format)s not found on Google Drive: %(fn)s",
-                                  format=self.settings['old_book_format'],
-                                  fn=data.name + "." + self.settings['old_book_format'].lower())
-                worker_db.session.close()
-                return self._handleError(error_message)
-
-        filename = self._convert_ebook_format()
-        if config.config_use_google_drive:
-            os.remove(self.file_path + '.' + self.settings['old_book_format'].lower())
-            if df_cover:
-                os.remove(os.path.join(config.config_calibre_dir, cur_book.path, "cover.jpg"))
-
-        if filename:
-            if config.config_use_google_drive:
-                # Upload files to gdrive
-                gdriveutils.updateGdriveCalibreFromLocal()
-                self._handleSuccess()
-            if self.ereader_mail:
-                # if we're sending to E-Reader after converting, create a one-off task and run it immediately
-                # todo: figure out how to incorporate this into the progress
-                try:
-                    EmailText = N_(u"%(book)s send to E-Reader", book=escape(self.title))                    
-                    for email in self.ereader_mail.split(','):
-                        email = strip_whitespaces(email)
-                        worker_thread.add(self.user, TaskEmail(self.settings['subject'],
-                                                               self.results["path"],
-                                                               filename,
-                                                               self.settings,
-                                                               email,
-                                                               EmailText,
-                                                               self.settings['body'],
-                                                               id=self.book_id,
-                                                               internal=True)
-                                          )
-                except Exception as ex:
-                    return self._handleError(str(ex))
-
-    def _convert_ebook_format(self):
-        error_message = None
-        local_db = db.CalibreDB(expire_on_commit=False, init=True)
-        file_path = self.file_path
-        book_id = self.book_id
-        format_old_ext = '.' + self.settings['old_book_format'].lower()
-        format_new_ext = '.' + self.settings['new_book_format'].lower()
-
-        # check to see if destination format already exists - or if book is in database
-        # if it does - mark the conversion task as complete and return a success
-        # this will allow to send to E-Reader workflow to continue to work
-        if os.path.isfile(file_path + format_new_ext) or\
-                local_db.get_book_format(self.book_id, self.settings['new_book_format']):
-            log.info("Book id %d already converted to %s", book_id, format_new_ext)
-            cur_book = local_db.get_book(book_id)
-            self.title = cur_book.title
-            self.results['path'] = cur_book.path
-            self.results['title'] = self.title
-            new_format = local_db.session.query(db.Data).filter(db.Data.book == book_id)\
-                .filter(db.Data.format == self.settings['new_book_format'].upper()).one_or_none()
-            if not new_format:
-                new_format = db.Data(name=os.path.basename(file_path),
-                                     book_format=self.settings['new_book_format'].upper(),
-                                     book=book_id, uncompressed_size=os.path.getsize(file_path + format_new_ext))
-                try:
-                    local_db.session.merge(new_format)
-                    local_db.session.commit()
-                except SQLAlchemyError as e:
-                    local_db.session.rollback()
-                    log.error("Database error: %s", e)
-                    local_db.session.close()
-                    self._handleError(N_("Oops! Database Error: %(error)s.", error=e))
-                    return
-                self._handleSuccess()
-                local_db.session.close()
-                return os.path.basename(file_path + format_new_ext)
-        else:
-            log.info("Book id %d - target format of %s does not exist. Moving forward with convert.",
-                     book_id,
-                     format_new_ext)
-
-        if config.config_kepubifypath and format_old_ext == '.epub' and format_new_ext == '.kepub':
-            check, error_message = self._convert_kepubify(file_path,
-                                                          format_old_ext,
-                                                          format_new_ext)
-        else:
-            # check if calibre converter-executable is existing
-            if not os.path.exists(config.config_converterpath):
-                self._handleError(N_("Calibre ebook-convert %(tool)s not found", tool=config.config_converterpath))
-                return
-            has_cover = local_db.get_book(book_id).has_cover
-            check, error_message = self._convert_calibre(file_path, format_old_ext, format_new_ext, has_cover)
-
-        if check == 0:
-            cur_book = local_db.get_book(book_id)
-            if os.path.isfile(file_path + format_new_ext):
-                new_format = local_db.session.query(db.Data).filter(db.Data.book == book_id) \
-                    .filter(db.Data.format == self.settings['new_book_format'].upper()).one_or_none()
-                if not new_format:
-                    new_format = db.Data(name=cur_book.data[0].name,
-                                         book_format=self.settings['new_book_format'].upper(),
-                                         book=book_id, uncompressed_size=os.path.getsize(file_path + format_new_ext))
-                    try:
-                        local_db.session.merge(new_format)
-                        local_db.session.commit()
-                        if self.settings['new_book_format'].upper() in ['KEPUB', 'EPUB', 'EPUB3']:
-                            ub_session = init_db_thread()
-                            remove_synced_book(book_id, True, ub_session)
-                            ub_session.close()
-                    except SQLAlchemyError as e:
-                        local_db.session.rollback()
-                        log.error("Database error: %s", e)
-                        local_db.session.close()
-                        self._handleError(error_message)
-                        return
-                self.results['path'] = cur_book.path
-                self.title = cur_book.title
-                self.results['title'] = self.title
-                if not config.config_use_google_drive:
-                    self._handleSuccess()
-                return os.path.basename(file_path + format_new_ext)
-            else:
-                error_message = N_('%(format)s format not found on disk', format=format_new_ext.upper())
-        local_db.session.close()
-        log.info("ebook converter failed with error while converting book")
-        if not error_message:
-            error_message = N_('Ebook converter failed with unknown error')
-        else:
-            log.error(error_message)
-        self._handleError(error_message)
-        return
-
-    def _convert_kepubify(self, file_path, format_old_ext, format_new_ext):
-        if config.config_embed_metadata and config.config_binariesdir:
-            tmp_dir, temp_file_name = helper.do_calibre_export(self.book_id, format_old_ext[1:])
-            filename = os.path.join(tmp_dir, temp_file_name + format_old_ext)
-            temp_file_path = tmp_dir
-        else:
-            filename = file_path + format_old_ext
-            temp_file_path = os.path.dirname(file_path)
-        quotes = [1, 3]
-        command = [config.config_kepubifypath, filename, '-o', temp_file_path, '-i']
-        try:
-            p = process_open(command, quotes)
-        except OSError as e:
-            return 1, N_("Kepubify-converter failed: %(error)s", error=e)
-        self.progress = 0.01
-        while True:
-            nextline = p.stdout.readlines()
-            nextline = [x.strip('\n') for x in nextline if x != '\n']
-            for line in nextline:
-                log.debug(line)
-            if p.poll() is not None:
-                break
-
-        # process returncode
-        check = p.returncode
-
-        # move file
-        if check == 0:
-            converted_file = glob(os.path.splitext(filename)[0] + "*.kepub.epub")
-            if len(converted_file) == 1:
-                copyfile(converted_file[0], (file_path + format_new_ext))
-                os.unlink(converted_file[0])
-            else:
-                return 1, N_("Converted file not found or more than one file in folder %(folder)s",
-                             folder=os.path.dirname(file_path))
-        return check, None
-
-    def _convert_calibre(self, file_path, format_old_ext, format_new_ext, has_cover):
-        path_tmp_opf = None
-        try:
-            # path_tmp_opf = self._embed_metadata()
-            if config.config_embed_metadata:
-                quotes = [5]
-                tmp_dir = get_temp_dir()
-                calibredb_binarypath = os.path.join(config.config_binariesdir, SUPPORTED_CALIBRE_BINARIES["calibredb"])
-                my_env = os.environ.copy()
-                if config.config_calibre_split:
-                    my_env['CALIBRE_OVERRIDE_DATABASE_PATH'] = os.path.join(config.config_calibre_dir, "metadata.db")
-                    library_path = config.config_calibre_split_dir
-                else:
-                    library_path = config.config_calibre_dir
-
-                opf_command = [calibredb_binarypath, 'show_metadata', '--as-opf', str(self.book_id),
-                               '--with-library', library_path]
-                p = process_open(opf_command, quotes, my_env)
-                p.wait()
-                check = p.returncode
-                calibre_traceback = p.stderr.readlines()
-                if check == 0:
-                    path_tmp_opf = os.path.join(tmp_dir, "metadata_" + str(uuid4()) + ".opf")
-                    with open(path_tmp_opf, 'w') as fd:
-                        copyfileobj(p.stdout, fd)
-                else:
-                    error_message = ""
-                    for ele in calibre_traceback:
-                        if not ele.startswith('Traceback') and not ele.startswith('  File'):
-                            error_message = N_("Calibre failed with error: %(error)s", error=ele)
-                    return check, error_message
-            quotes = [1, 2, 4, 6]
-            command = [config.config_converterpath, (file_path + format_old_ext),
-                       (file_path + format_new_ext)]
-            if config.config_embed_metadata:
-                command.extend(['--from-opf', path_tmp_opf])
-            if has_cover:
-                command.extend(['--cover', os.path.join(os.path.dirname(file_path), 'cover.jpg')])
-            quotes_index = 3
-            if config.config_calibre:
-                parameters = config.config_calibre.split(" ")
-                for param in parameters:
-                    command.append(param)
-                    quotes.append(quotes_index)
-                    quotes_index += 1
-
-            p = process_open(command, quotes, newlines=False)
-        except OSError as e:
-            return 1, N_("Ebook-converter failed: %(error)s", error=e)
-
-        while p.poll() is None:
-            nextline = p.stdout.readline()
-            if isinstance(nextline, bytes):
-                nextline = nextline.decode('utf-8', errors="ignore").strip('\r\n')
-            if nextline:
-                log.debug(nextline)
-            # parse progress string from calibre-converter
-            progress = re.search(r"(\d+)%\s.*", nextline)
-            if progress:
-                self.progress = int(progress.group(1)) / 100
-                if config.config_use_google_drive:
-                    self.progress *= 0.9
-
-        # process returncode
-        check = p.returncode
-        calibre_traceback = p.stderr.readlines()
-        error_message = ""
-        for ele in calibre_traceback:
-            ele = ele.decode('utf-8', errors="ignore").strip('\n')
-            log.debug(ele)
-            if not ele.startswith('Traceback') and not ele.startswith('  File'):
-                error_message = N_("Calibre failed with error: %(error)s", error=ele)
-        return check, error_message
-
-    @property
-    def name(self):
-        return N_("Convert")
-
-    def __str__(self):
-        if self.ereader_mail:
-            return "Convert Book {} and mail it to {}".format(self.book_id, self.ereader_mail)
-        else:
-            return "Convert Book {}".format(self.book_id)
-
-    @property
-    def is_cancellable(self):
-        return False
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2020 pwr
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import re
+import glob
+from shutil import copyfile, copyfileobj
+from markupsafe import escape
+from time import time
+from uuid import uuid4
+
+from sqlalchemy.exc import SQLAlchemyError
+from flask_babel import lazy_gettext as N_
+
+from cps.services.worker import CalibreTask
+from cps import db
+from cps import logger, config
+from cps.subproc_wrapper import process_open
+from flask_babel import gettext as _
+from cps.kobo_sync_status import remove_synced_book
+from cps.ub import init_db_thread
+from cps.file_helper import get_temp_dir
+
+from cps.tasks.mail import TaskEmail
+from cps import gdriveutils, helper
+from cps.constants import SUPPORTED_CALIBRE_BINARIES
+from cps.string_helper import strip_whitespaces
+
+log = logger.create()
+
+current_milli_time = lambda: int(round(time() * 1000))
+
+
+class TaskConvert(CalibreTask):
+    def __init__(self, file_path, book_id, task_message, settings, ereader_mail, user=None):
+        super(TaskConvert, self).__init__(task_message)
+        self.worker_thread = None
+        self.file_path = file_path
+        self.book_id = book_id
+        self.title = ""
+        self.settings = settings
+        self.ereader_mail = ereader_mail
+        self.user = user
+
+        self.results = dict()
+
+    def run(self, worker_thread):
+        self.worker_thread = worker_thread
+        if config.config_use_google_drive:
+            worker_db = db.CalibreDB(expire_on_commit=False, init=True)
+            cur_book = worker_db.get_book(self.book_id)
+            self.title = cur_book.title
+            data = worker_db.get_book_format(self.book_id, self.settings['old_book_format'])
+            df = gdriveutils.getFileFromEbooksFolder(cur_book.path,
+                                                     data.name + "." + self.settings['old_book_format'].lower())
+            df_cover = gdriveutils.getFileFromEbooksFolder(cur_book.path, "cover.jpg")
+            if df:
+                datafile_cover = None
+                datafile = os.path.join(config.get_book_path(),
+                                        cur_book.path,
+                                        data.name + "." + self.settings['old_book_format'].lower())
+                if df_cover:
+                    datafile_cover = os.path.join(config.get_book_path(),
+                                                  cur_book.path, "cover.jpg")
+                if not os.path.exists(os.path.join(config.get_book_path(), cur_book.path)):
+                    os.makedirs(os.path.join(config.get_book_path(), cur_book.path))
+                df.GetContentFile(datafile)
+                if df_cover:
+                    df_cover.GetContentFile(datafile_cover)
+                worker_db.session.close()
+            else:
+                # ToDo Include cover in error handling
+                error_message = _("%(format)s not found on Google Drive: %(fn)s",
+                                  format=self.settings['old_book_format'],
+                                  fn=data.name + "." + self.settings['old_book_format'].lower())
+                worker_db.session.close()
+                return self._handleError(error_message)
+
+        filename = self._convert_ebook_format()
+        if config.config_use_google_drive:
+            os.remove(self.file_path + '.' + self.settings['old_book_format'].lower())
+            if df_cover:
+                os.remove(os.path.join(config.config_calibre_dir, cur_book.path, "cover.jpg"))
+
+        if filename:
+            if config.config_use_google_drive:
+                # Upload files to gdrive
+                gdriveutils.updateGdriveCalibreFromLocal()
+                self._handleSuccess()
+            if self.ereader_mail:
+                # if we're sending to E-Reader after converting, create a one-off task and run it immediately
+                # todo: figure out how to incorporate this into the progress
+                try:
+                    EmailText = N_(u"%(book)s send to E-Reader", book=escape(self.title))                    
+                    for email in self.ereader_mail.split(','):
+                        email = strip_whitespaces(email)
+                        worker_thread.add(self.user, TaskEmail(self.settings['subject'],
+                                                               self.results["path"],
+                                                               filename,
+                                                               self.settings,
+                                                               email,
+                                                               EmailText,
+                                                               self.settings['body'],
+                                                               id=self.book_id,
+                                                               internal=True)
+                                          )
+                except Exception as ex:
+                    return self._handleError(str(ex))
+
+    def _convert_ebook_format(self):
+        error_message = None
+        local_db = db.CalibreDB(expire_on_commit=False, init=True)
+        file_path = self.file_path
+        book_id = self.book_id
+        format_old_ext = '.' + self.settings['old_book_format'].lower()
+        format_new_ext = '.' + self.settings['new_book_format'].lower()
+
+        # check to see if destination format already exists - or if book is in database
+        # if it does - mark the conversion task as complete and return a success
+        # this will allow to send to E-Reader workflow to continue to work
+        if os.path.isfile(file_path + format_new_ext) or\
+                local_db.get_book_format(self.book_id, self.settings['new_book_format']):
+            log.info("Book id %d already converted to %s", book_id, format_new_ext)
+            cur_book = local_db.get_book(book_id)
+            self.title = cur_book.title
+            self.results['path'] = cur_book.path
+            self.results['title'] = self.title
+            new_format = local_db.session.query(db.Data).filter(db.Data.book == book_id)\
+                .filter(db.Data.format == self.settings['new_book_format'].upper()).one_or_none()
+            if not new_format:
+                new_format = db.Data(name=os.path.basename(file_path),
+                                     book_format=self.settings['new_book_format'].upper(),
+                                     book=book_id, uncompressed_size=os.path.getsize(file_path + format_new_ext))
+                try:
+                    local_db.session.merge(new_format)
+                    local_db.session.commit()
+                except SQLAlchemyError as e:
+                    local_db.session.rollback()
+                    log.error("Database error: %s", e)
+                    local_db.session.close()
+                    self._handleError(N_("Oops! Database Error: %(error)s.", error=e))
+                    return
+                self._handleSuccess()
+                local_db.session.close()
+                return os.path.basename(file_path + format_new_ext)
+        else:
+            log.info("Book id %d - target format of %s does not exist. Moving forward with convert.",
+                     book_id,
+                     format_new_ext)
+
+        if config.config_kepubifypath and format_old_ext == '.epub' and format_new_ext == '.kepub':
+            check, error_message = self._convert_kepubify(file_path,
+                                                          format_old_ext,
+                                                          format_new_ext)
+        else:
+            # check if calibre converter-executable is existing
+            if not os.path.exists(config.config_converterpath):
+                self._handleError(N_("Calibre ebook-convert %(tool)s not found", tool=config.config_converterpath))
+                return
+            has_cover = local_db.get_book(book_id).has_cover
+            check, error_message = self._convert_calibre(file_path, format_old_ext, format_new_ext, has_cover)
+
+        if check == 0:
+            cur_book = local_db.get_book(book_id)
+            if os.path.isfile(file_path + format_new_ext):
+                new_format = local_db.session.query(db.Data).filter(db.Data.book == book_id) \
+                    .filter(db.Data.format == self.settings['new_book_format'].upper()).one_or_none()
+                if not new_format:
+                    new_format = db.Data(name=cur_book.data[0].name,
+                                         book_format=self.settings['new_book_format'].upper(),
+                                         book=book_id, uncompressed_size=os.path.getsize(file_path + format_new_ext))
+                    try:
+                        local_db.session.merge(new_format)
+                        local_db.session.commit()
+                        if self.settings['new_book_format'].upper() in ['KEPUB', 'EPUB', 'EPUB3']:
+                            ub_session = init_db_thread()
+                            remove_synced_book(book_id, True, ub_session)
+                            ub_session.close()
+                    except SQLAlchemyError as e:
+                        local_db.session.rollback()
+                        log.error("Database error: %s", e)
+                        local_db.session.close()
+                        self._handleError(error_message)
+                        return
+                self.results['path'] = cur_book.path
+                self.title = cur_book.title
+                self.results['title'] = self.title
+                if not config.config_use_google_drive:
+                    self._handleSuccess()
+                return os.path.basename(file_path + format_new_ext)
+            else:
+                error_message = N_('%(format)s format not found on disk', format=format_new_ext.upper())
+        local_db.session.close()
+        log.info("ebook converter failed with error while converting book")
+        if not error_message:
+            error_message = N_('Ebook converter failed with unknown error')
+        else:
+            log.error(error_message)
+        self._handleError(error_message)
+        return
+
+    def _convert_kepubify(self, file_path, format_old_ext, format_new_ext):
+        if config.config_embed_metadata and config.config_binariesdir:
+            tmp_dir, temp_file_name = helper.do_calibre_export(self.book_id, format_old_ext[1:])
+            filename = os.path.join(tmp_dir, temp_file_name + format_old_ext)
+            temp_file_path = tmp_dir
+        else:
+            filename = file_path + format_old_ext
+            temp_file_path = os.path.dirname(file_path)
+        quotes = [1, 3]
+        command = [config.config_kepubifypath, filename, '-o', temp_file_path, '-i']
+        try:
+            p = process_open(command, quotes)
+        except OSError as e:
+            return 1, N_("Kepubify-converter failed: %(error)s", error=e)
+        self.progress = 0.01
+        while True:
+            nextline = p.stdout.readlines()
+            nextline = [x.strip('\n') for x in nextline if x != '\n']
+            for line in nextline:
+                log.debug(line)
+            if p.poll() is not None:
+                break
+
+        # process returncode
+        check = p.returncode
+
+        # move file
+        if check == 0:
+            converted_file = glob.glob(glob.escape(os.path.splitext(filename)[0]) + "*.kepub.epub")
+            if len(converted_file) == 1:
+                copyfile(converted_file[0], (file_path + format_new_ext))
+                os.unlink(converted_file[0])
+            else:
+                return 1, N_("Converted file not found or more than one file in folder %(folder)s",
+                             folder=os.path.dirname(file_path))
+        return check, None
+
+    def _convert_calibre(self, file_path, format_old_ext, format_new_ext, has_cover):
+        path_tmp_opf = None
+        try:
+            # path_tmp_opf = self._embed_metadata()
+            if config.config_embed_metadata:
+                quotes = [5]
+                tmp_dir = get_temp_dir()
+                calibredb_binarypath = os.path.join(config.config_binariesdir, SUPPORTED_CALIBRE_BINARIES["calibredb"])
+                my_env = os.environ.copy()
+                if config.config_calibre_split:
+                    my_env['CALIBRE_OVERRIDE_DATABASE_PATH'] = os.path.join(config.config_calibre_dir, "metadata.db")
+                    library_path = config.config_calibre_split_dir
+                else:
+                    library_path = config.config_calibre_dir
+
+                opf_command = [calibredb_binarypath, 'show_metadata', '--as-opf', str(self.book_id),
+                               '--with-library', library_path]
+                p = process_open(opf_command, quotes, my_env, newlines=False)
+                lines = list()
+                while p.poll() is None:
+                    lines.append(p.stdout.readline())
+                check = p.returncode
+                calibre_traceback = p.stderr.readlines()
+                if check == 0:
+                    path_tmp_opf = os.path.join(tmp_dir, "metadata_" + str(uuid4()) + ".opf")
+                    with open(path_tmp_opf, 'wb') as fd:
+                        fd.write(b''.join(lines))
+                else:
+                    error_message = ""
+                    for ele in calibre_traceback:
+                        if not ele.startswith('Traceback') and not ele.startswith('  File'):
+                            error_message = N_("Calibre failed with error: %(error)s", error=ele)
+                    return check, error_message
+            quotes = [1, 2]
+            quotes_index = 3
+            command = [config.config_converterpath, (file_path + format_old_ext),
+                       (file_path + format_new_ext)]
+            if config.config_embed_metadata:
+                quotes.append(4)
+                quotes_index = 5
+                command.extend(['--from-opf', path_tmp_opf])
+                if has_cover:
+                    quotes.append(6)
+                    command.extend(['--cover', os.path.join(os.path.dirname(file_path), 'cover.jpg')])
+                    quotes_index = 7
+            if config.config_calibre:
+                parameters = re.findall(r"(--[\w-]+)(?:(\s(?:(\".+\")|(?:.+?)))(?:\s|$))?",
+                                        config.config_calibre, re.IGNORECASE | re.UNICODE)
+                if parameters:
+                    for param in parameters:
+                        command.append(strip_whitespaces(param[0]))
+                        quotes_index += 1
+                        if param[1] != "":
+                            parsed = strip_whitespaces(param[1]).strip("\"")
+                            command.append(parsed)
+                            quotes.append(quotes_index)
+                            quotes_index += 1
+            p = process_open(command, quotes, newlines=False)
+        except OSError as e:
+            return 1, N_("Ebook-converter failed: %(error)s", error=e)
+
+        while p.poll() is None:
+            nextline = p.stdout.readline()
+            if isinstance(nextline, bytes):
+                nextline = nextline.decode('utf-8', errors="ignore").strip('\r\n')
+            if nextline:
+                log.debug(nextline)
+            # parse progress string from calibre-converter
+            progress = re.search(r"(\d+)%\s.*", nextline)
+            if progress:
+                self.progress = int(progress.group(1)) / 100
+                if config.config_use_google_drive:
+                    self.progress *= 0.9
+
+        # process returncode
+        check = p.returncode
+        calibre_traceback = p.stderr.readlines()
+        error_message = ""
+        for ele in calibre_traceback:
+            ele = ele.decode('utf-8', errors="ignore").strip('\n')
+            log.debug(ele)
+            if not ele.startswith('Traceback') and not ele.startswith('  File'):
+                error_message = N_("Calibre failed with error: %(error)s", error=ele)
+        return check, error_message
+
+    @property
+    def name(self):
+        return N_("Convert")
+
+    def __str__(self):
+        if self.ereader_mail:
+            return "Convert Book {} and mail it to {}".format(self.book_id, self.ereader_mail)
+        else:
+            return "Convert Book {}".format(self.book_id)
+
+    @property
+    def is_cancellable(self):
+        return False

cps/templates/author.html

@@ -71,7 +71,7 @@
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.Books.series[0].id )}}">
             {{entry.Books.series[0].name}}
           </a>
-          ({{entry.Books.series_index|formatseriesindex}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
         {% if entry.Books.ratings.__len__() > 0 %}
@@ -122,7 +122,7 @@
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.series[0].id )}}">
             {{entry.series[0].name}}
           </a>
-          ({{entry.series_index|formatseriesindex}})
+          ({{entry.series_index|formatfloat(2)}})
         </p>
         {% endif %}
         <div class="rating">

cps/templates/basic_detail.html

@@ -1,81 +0,0 @@
-{% extends "basic_layout.html" %}
-
-{% block body %}
-<div>
-
-<h2 id="title">{{ entry.title }}</h2>
-<div>
-  {% for author in entry.ordered_authors %}
-    <p>{{ author.name.replace("|",",") }}</p>
-  {% endfor %}
-</div>
-
-<div class="cover">
-    <img title="{{ entry.title }}" src="{{ url_for('web.get_cover', book_id=entry.id, resolution='og', c=entry|last_modified) }}"/>
-</div>
-
-{% if current_user.role_download() %}
-  {% if entry.data|length %}
-    <div>
-      <h2>Download</h2>
-        {% for format in entry.data %}
-          <p>
-            <a href="{{ url_for('web.download_link', book_id=entry.id, book_format=format.format|lower, anyname=entry.id|string+'.'+format.format|lower) }}">
-              {{ format.format }} ({{ format.uncompressed_size|filesizeformat }})</a>
-          </p>
-        {% endfor %}
-    </div>
-  {% endif %}
-{% endif %}
-
-<h2>Details</h2>
-
-{% if entry.series|length > 0 %}
-  <p>{{ _("Book %(index)s of %(range)s", index=entry.series_index | formatfloat(2), range=(entry.series[0].name)|safe) }}</p>
-{% endif %}
-
-{% if entry.languages|length > 0 %}
-  <div>
-    <p>
-      <span>
-        {{_('Language')}}: {% for language in entry.languages %}{{language.language_name}}{% if not loop.last %}, {% endif %}{% endfor %}
-      </span>
-    </p>
-  </div>
-{% endif %}
-
-{% if entry.identifiers|length > 0 %}
-  <div>
-    <p>
-      <span></span>
-      {% for identifier in entry.identifiers %}
-          <p>{{ identifier.format_type() }}: {{ identifier|escape }}</p>
-      {% endfor %}
-    </p>
-  </div>
-{% endif %}
-
-{% if entry.publishers|length > 0 %}
-  <div>
-    <p>
-      <span>{{ _('Publisher') }}:
-        <span>{{ entry.publishers[0].name }}</span>
-      </span>
-    </p>
-  </div>
-{% endif %}
-
-{% if (entry.pubdate|string)[:10] != '0101-01-01' %}
-  <div>
-    <p>{{ _('Published') }}: {{ entry.pubdate|formatdate }} </p>
-  </div>
-{% endif %}
-
-{% if entry.comments|length > 0 and entry.comments[0].text|length > 0 %}
-    <div>
-        <h2 id="decription">{{ _('Description:') }}</h2>
-        {{ entry.comments[0].text|safe }}
-    </div>
-{% endif %}
-</div>
-{% endblock %}

cps/templates/basic_index.html

@@ -1,32 +0,0 @@
-{% extends "basic_layout.html" %}
-{% block body %}
-
-<div class="pagination">
-  <div>
-    {% if pagination.has_prev %}
-      <a href="{{ (pagination.page - 1)|url_for_other_page }}">&laquo; {{_('Previous')}}</a>
-    {% endif %}
-  </div>
-  <div>
-    {% if pagination.has_next %}
-      <a href="{{ (pagination.page + 1)|url_for_other_page }}">{{_('Next')}} &raquo;</a>
-    {% endif %}
-  </div>
-</div>
-
-{% if entries|length < 1 %}
-  <p>{{_('No Results Found')}}</p>
-{% endif %}
-
-{% for entry in entries %}
-  {% if entry.Books.authors %}
-    {% set author = entry.Books.authors[0].name.replace('|',',')|shortentitle(30) %}
-  {% else %}
-    {% set author = '' %}
-  {% endif %}
-  <a href="{{ url_for('basic.show_book', book_id=entry.Books.id) }}">
-    <p class="listing" title="{{entry.Books.title}}">{{ author }} - {{entry.Books.title|shortentitle}}</p>
-  </a>
-{% endfor %}
-
-{% endblock %}

cps/templates/basic_layout.html

@@ -1,147 +0,0 @@
-<!DOCTYPE html>
-<html lang="{{ current_user.locale }}">
-
-<head>
-<title>{{instance}} | {{title}}</title>
-<meta charset="utf-8">
-<meta name='viewport' content='initial-scale=1,maximum-scale=5,user-scalable=no' />
-<style>
-body {
-  margin: 0;
-}
-
-nav {
-  height: 75px;
-  padding: 5px 20px;
-  width: 100%;
-  display: table;
-  box-sizing: border-box;
-  border-bottom: 1px solid black;
-}
-
-nav > * {
-  display: inline-block;
-  display: table-cell;
-  vertical-align: middle;
-  float: none;
-  text-align: center;
-  width: auto;
-}
-
-nav > *:first-child {
-  text-align: left;
-  width: 1%;
-}
-
-nav > *:last-child {
-  text-align: right;
-  width: 1%;
-}
-
-nav > a {
-  color: black;
-  margin: 0 20px;
-}
-
-.search {
-  margin: auto auto;
-}
-
-form > input {
-  width: 18ch;
-  padding-left: 4px;
-}
-
-form > * {
-  height: 50px;
-  background-color: white;
-  border-radius: 0;
-  border: 1px solid #ccc;
-  padding: 0;
-  margin: 0;
-  display: inline-block;
-  vertical-align: top;
-}
-
-form > span {
-  margin-left: -5px;
-}
-
-button {
-  border: none;
-  padding: 0 10px;
-  margin: 0;
-  width: 160px;
-  height: 100%;
-  background-color: white;
-}
-
-.body {
-  padding: 5px 20px;
-}
-
-a {
-  color: black;
-}
-
-img {
-  width: 150px;
-  height: 250px;
-  object-fit: cover;
-}
-
-.listing {
-  white-space: nowrap;
-  overflow: hidden;
-  text-overflow: ellipsis;
-  margin-right: 20px;
-}
-
-.pagination {
-  padding: 10px 0;
-  height: 20px;
-  font-weight: 700;
-}
-
-.pagination > div {
-  float: left;
-}
-
-.pagination > div:last-child {
-  float: right;
-}
-</style>
-</head>
-
-<body>
-  <div>
-    <div>
-      {% if current_user.is_authenticated or g.allow_anonymous %}
-        <nav>
-          <a href="/basic">
-            <span><h1>{{_('Home')}}</h1></span>
-          </a>
-          <div class="search">
-            <form role="search" action="{{url_for('basic.index')}}" method="GET">
-              <input type="text" id="query" name="query" placeholder="{{_('Search Library')}}" value="{{searchterm}}">
-              <span>
-                <button type="submit" id="query_submit">{{_('Search')}}</button>
-              </span>
-            </form>
-          </div>
-          {% if not current_user.is_anonymous %}
-            <a href="{{url_for('web.logout')}}">
-              <span>{{_('Logout')}}</span>
-            </a>
-          {% endif %}
-        </nav>
-      {% endif %}
-    </div>
-  </div>
-  <div class="body">
-    {% block body %}
-    {% endblock %}
-  </div>
-</body>
-
-</html>

cps/templates/book_edit.html

@@ -108,7 +108,7 @@
     </div>
     <div class="form-group">
       <label for="series_index">{{_('Series ID')}}</label>
-      <input type="number" step="0.01" min="0" placeholder="1" class="form-control" name="series_index" id="series_index" value="{{book.series_index}}">
+      <input type="number" step="0.01" min="0" placeholder="1" class="form-control" name="series_index" id="series_index" value="{{book.series_index|formatfloat(2)}}">
     </div>
     <div class="form-group">
       <label for="rating">{{_('Rating')}}</label>

cps/templates/book_table.html

@@ -66,7 +66,7 @@
             {{ text_table_row('authors', _('Enter Authors'),_('Authors'), true, true) }}
             {{ text_table_row('tags', _('Enter Categories'),_('Categories'), false, true) }}
             {{ text_table_row('series', _('Enter Series'),_('Series'), false, true) }}
-            <th data-field="series_index" id="series_index" data-visible="{{visiblility.get('series_index')}}" data-edit-validate="{{ _('This Field is Required') }}" data-sortable="true" {% if current_user.role_edit() %} data-editable-type="number" data-editable-placeholder="1" data-editable-step="0.01" data-editable-min="0" data-editable-url="{{ url_for('edit-book.edit_list_book', param='series_index')}}" data-edit="true" data-editable-title="{{_('Enter Title')}}"{% endif %}>{{_('Series Index')}}</th>
+            <th data-field="series_index" id="series_index" data-visible="{{visiblility.get('series_index')}}"  data-formatter="seriesIndexFormatter" data-edit-validate="{{ _('This Field is Required') }}" data-sortable="true" {% if current_user.role_edit() %} data-editable-type="number" data-editable-placeholder="1" data-editable-step="0.01" data-editable-min="0" data-editable-url="{{ url_for('edit-book.edit_list_book', param='series_index')}}" data-edit="true" data-editable-title="{{_('Enter Title')}}"{% endif %}>{{_('Series Index')}}</th>
             {{ text_table_row('languages', _('Enter Languages'),_('Languages'), false, true) }}
             <!--th data-field="pubdate" data-type="date" data-visible="{{visiblility.get('pubdate')}}" data-viewformat="dd.mm.yyyy" id="pubdate" data-sortable="true">{{_('Publishing Date')}}</th-->
             {{ text_table_row('publishers', _('Enter Publishers'),_('Publishers'), false, true) }}

cps/templates/detail.html

@@ -147,7 +147,7 @@
                 </div>
             {% endif %}
             {% if entry.series|length > 0 %}
-                <p>{{ _("Book %(index)s of %(range)s", index=entry.series_index | formatfloat(2), range=(url_for('web.books_list', data='series', sort_param='stored', book_id=entry.series[0].id)|escapedlink(entry.series[0].name))|safe) }}</p>
+                <p>{{ _("Book %(index)s of %(range)s", index=entry.series_index|formatfloat(2), range=(url_for('web.books_list', data='series', sort_param='stored', book_id=entry.series[0].id)|escapedlink(entry.series[0].name))|safe) }}</p>
 
             {% endif %}
 

cps/templates/index.html

@@ -42,7 +42,7 @@
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.Books.series[0].id )}}">
             {{entry.Books.series[0].name}}
           </a>
-          ({{entry.Books.series_index|formatseriesindex}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
         {% if entry.Books.ratings.__len__() > 0 %}
@@ -132,7 +132,7 @@
         {% else %}
             <span>{{entry.Books.series[0].name}}</span>
         {% endif %}
-          ({{entry.Books.series_index|formatseriesindex}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
         {% if entry.Books.ratings.__len__() > 0 %}

cps/templates/layout.html

@@ -37,13 +37,11 @@
           <a class="navbar-brand" href="{{url_for('web.index')}}">{{instance}}</a>
         </div>
         {% if g.current_theme == 1 %}
-            <div class="home-btn"><a class="home-btn-tooltip" href="{{url_for("web.index", page=1)}}" data-toggle="tooltip" title="" data-placement="bottom" data-original-title="Home"></a></div>
+            <div class="home-btn"><a class="home-btn-tooltip" href="{{url_for("web.index",page=1)}}" data-toggle="tooltip" title="" data-placement="bottom" data-original-title="Home"></a></div>
             <div class="plexBack"><a href="{{url_for('web.index')}}"></a></div>
         {% endif %}
         {% if current_user.is_authenticated or g.allow_anonymous %}
-          <a href="{{url_for('basic.index')}}" class="navbar-form navbar-left btn btn-default" id="basic"><span class="glyphicon glyphicon-phone"></span><span>{{_('Simple view')}}</span></a>
-          <!--# margin 0, padding 15, background color-->
-           <form class="navbar-form navbar-left" role="search" action="{{url_for('search.simple_search')}}" method="GET">
+          <form class="navbar-form navbar-left" role="search" action="{{url_for('search.simple_search')}}" method="GET">
             <div class="form-group input-group input-group-sm">
               <label for="query" class="sr-only">{{_('Search')}}</label>
               <input type="text" class="form-control" id="query" name="query" placeholder="{{_('Search Library')}}" value="{{searchterm}}">

cps/templates/listenmp3.html

@@ -59,7 +59,7 @@
         </div>
       {% endif %}
       {% if entry.series|length > 0 %}
-          <p>{{_("Book %(index)s of %(range)s", index=entry.series_index | formatfloat(2), range=(url_for('web.books_list', data='series', sort_param='stored', book_id=entry.series[0].id)|escapedlink(entry.series[0].name))|safe)}}</p>
+          <p>{{_("Book %(index)s of %(range)s", index=entry.series_index|formatfloat(2), range=(url_for('web.books_list', data='series', sort_param='stored', book_id=entry.series[0].id)|escapedlink(entry.series[0].name))|safe)}}</p>
 
       {% endif %}
 

cps/templates/read.html

@@ -135,19 +135,23 @@
         window.themes = {
             "darkTheme": {
                 "bgColor": "#202124",
-                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}"
+                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}",
+                "title-color": "#fff"
             },
             "lightTheme": {
                 "bgColor": "white",
-                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}"
+                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}",
+                "title-color": "#4f4f4f"
             },
             "sepiaTheme": {
                 "bgColor": "#ece1ca",
-                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}"
+                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}",
+                "title-color": "#4f4f4f"
             },
             "blackTheme": {
                 "bgColor": "black",
-                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}"
+                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}",
+                "title-color": "#fff"
             },
         };
 
@@ -170,6 +174,7 @@
 
             // Apply theme to rest of the page. 
             document.getElementById("main").style.backgroundColor = themes[id]["bgColor"];
+            document.getElementById("titlebar").style.color = themes[id]["title-color"] || "#fff";
         }
 
         // font size settings logic

cps/templates/search.html

@@ -82,7 +82,7 @@
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.Books.series[0].id )}}">
             {{entry.Books.series[0].name}}
           </a>
-          ({{entry.Books.series_index|formatseriesindex}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
 

cps/templates/shelf.html

@@ -70,7 +70,7 @@
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.Books.series[0].id )}}">
             {{entry.Books.series[0].name}}
           </a>
-          ({{entry.Books.series_index|formatseriesindex}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
         {% if entry.Books.ratings.__len__() > 0 %}

cps/templates/shelf_order.html

@@ -20,7 +20,7 @@
                   {{entry['Books']['title']}}
                   {% if entry['Books']['series']|length > 0 %}
                     <br>
-                    {{entry['Books']['series_index']}} - {{entry['Books']['series'][0].name}}
+                    {{entry['Books']['series_index']|formatfloat(2)}} - {{entry['Books']['series'][0].name}}
                   {% endif %}
                   <br>
                   {% for author in entry['Books']['author'] %}

cps/templates/shelfdown.html

@@ -45,7 +45,7 @@
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.Books.series[0].id )}}">
             {{entry.Books.series[0].name}}
           </a>
-          ({{entry.Books.series_index}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
       </div>