Update Version

Update teststatus

MANIFEST.in

@@ -1 +1,3 @@
 graft src/calibreweb
+global-exclude __pycache__
+global-exclude *.pyc

README.md

@@ -1,10 +1,3 @@
-# Short Notice from the maintainer 
-
-After 6 years of more or less intensive programming on Calibre-Web, I need a break. 
-The last few months, maintaining Calibre-Web has felt more like work than a hobby. I felt pressured and teased by people to solve "their" problems and merge PRs for "their" Calibre-Web. 
-I have turned off all notifications from Github/Discord and will now concentrate undisturbed on the development of “my” Calibre-Web over the next few weeks/months.  
-I will look into the issues and maybe also the PRs from time to time, but don't expect a quick response from me.
-
 # Calibre-Web
 
 Calibre-Web is a web app that offers a clean and intuitive interface for browsing, reading, and downloading eBooks using a valid [Calibre](https://calibre-ebook.com) database.
@@ -26,13 +19,13 @@ Calibre-Web is a web app that offers a clean and intuitive interface for browsin
    - [Quick start](#quick-start)
    - [Requirements](#requirements)
 4. [Docker Images](#docker-images)
-5. [Contributor Recognition](#contributor-recognition)
-6. [Contact](#contact)
-7. [Contributing to Calibre-Web](#contributing-to-calibre-web)
+5. [Troubleshooting](#troubleshooting)
+6. [Contributor Recognition](#contributor-recognition)
+7. [Contact](#contact)
+8. [Contributing to Calibre-Web](#contributing-to-calibre-web)
 
 </details>
 
-
 *This software is a fork of [library](https://github.com/mutschler/calibreserver) and licensed under the GPL v3 License.*
 
 ![Main screen](https://github.com/janeczku/calibre-web/wiki/images/main_screen.png)
@@ -64,52 +57,103 @@ Calibre-Web is a web app that offers a clean and intuitive interface for browsin
 
 ## Installation
 
-#### Installation via pip (recommended)
-1. Create a virtual environment for Calibre-Web to avoid conflicts with existing Python dependencies
-2. Install Calibre-Web via pip: `pip install calibreweb` (or `pip3` depending on your OS/distro)
-3. Install optional features via pip as needed, see [this page](https://github.com/janeczku/calibre-web/wiki/Dependencies-in-Calibre-Web-Linux-and-Windows) for details
-4. Start Calibre-Web by typing `cps`
+### Installation via pip (recommended)
 
-*Note: Raspberry Pi OS users may encounter issues during installation. If so, please update pip (`./venv/bin/python3 -m pip install --upgrade pip`) and/or install cargo (`sudo apt install cargo`) before retrying the installation.*
+1. **Create a virtual environment**: It’s essential to isolate your Calibre-Web installation to avoid dependency conflicts. You can create a virtual environment by running:
+   ```
+   python3 -m venv calibre-web-env
+   ```
+2. **Activate the virtual environment**:
+   ```
+   source calibre-web-env/bin/activate
+   ```
+3. **Install Calibre-Web**: Use pip to install the application:
+   ```
+   pip install calibreweb
+   ```
+4. **Install optional features**: For additional functionality, you may need to install optional features. Refer to [this page](https://github.com/janeczku/calibre-web/wiki/Dependencies-in-Calibre-Web-Linux-and-Windows) for details on what can be installed.
+5. **Start Calibre-Web**: After installation, you can start the application with:
+   ```
+   cps
+   ```
 
-Refer to the Wiki for additional installation examples: [manual installation](https://github.com/janeczku/calibre-web/wiki/Manual-installation), [Linux Mint](https://github.com/janeczku/calibre-web/wiki/How-To:-Install-Calibre-Web-in-Linux-Mint-19-or-20), [Cloud Provider](https://github.com/janeczku/calibre-web/wiki/How-To:-Install-Calibre-Web-on-a-Cloud-Provider).
+*Note: Users of Raspberry Pi OS may encounter installation issues. If you do, try upgrading pip and/or installing cargo as follows:*
+   ```
+   ./venv/bin/python3 -m pip install --upgrade pip
+   sudo apt install cargo
+   ```
+
+### Important Links
+- For additional installation examples, check the following:
+   - [Manual installation](https://github.com/janeczku/calibre-web/wiki/Manual-installation)
+   - [Linux Mint installation](https://github.com/janeczku/calibre-web/wiki/How-To:-Install-Calibre-Web-in-Linux-Mint-19-or-20)
+   - [Cloud Provider setup](https://github.com/janeczku/calibre-web/wiki/How-To:-Install-Calibre-Web-on-a-Cloud-Provider)
 
 ## Quick Start
 
-1. Open your browser and navigate to `http://localhost:8083` or `http://localhost:8083/opds` for the OPDS catalog
-2. Log in with the default admin credentials
-3. If you don't have a Calibre database, you can use [this database](https://github.com/janeczku/calibre-web/raw/master/library/metadata.db) (move it out of the Calibre-Web folder to prevent overwriting during updates)
-4. Set `Location of Calibre database` to the path of the folder containing your Calibre library (metadata.db) and click "Save"
-5. Optionally, use Google Drive to host your Calibre library by following the [Google Drive integration guide](https://github.com/janeczku/calibre-web/wiki/G-Drive-Setup#using-google-drive-integration)
-6. Configure your Calibre-Web instance via the admin page, referring to the [Basic Configuration](https://github.com/janeczku/calibre-web/wiki/Configuration#basic-configuration) and [UI Configuration](https://github.com/janeczku/calibre-web/wiki/Configuration#ui-configuration) guides
-
-#### Default Admin Login:
-- **Username:** admin
-- **Password:** admin123
+1. **Access Calibre-Web**: Open your browser and navigate to:
+   ```
+   http://localhost:8083
+   ```
+   or for the OPDS catalog:
+   ```
+   http://localhost:8083/opds
+   ```
+2. **Log in**: Use the default admin credentials:
+   - **Username:** admin
+   - **Password:** admin123
+3. **Database Setup**: If you do not have a Calibre database, download a sample from:
+   ```
+   https://github.com/janeczku/calibre-web/raw/master/library/metadata.db
+   ```
+   Move it out of the Calibre-Web folder to avoid overwriting during updates.
+4. **Configure Calibre Database**: In the admin interface, set the `Location of Calibre database` to the path of the folder containing your Calibre library (where `metadata.db` is located) and click "Save".
+5. **Google Drive Integration**: For hosting your Calibre library on Google Drive, refer to the [Google Drive integration guide](https://github.com/janeczku/calibre-web/wiki/G-Drive-Setup#using-google-drive-integration).
+6. **Admin Configuration**: Configure your instance via the admin page, referring to the [Basic Configuration](https://github.com/janeczku/calibre-web/wiki/Configuration#basic-configuration) and [UI Configuration](https://github.com/janeczku/calibre-web/wiki/Configuration#ui-configuration) guides.
 
 ## Requirements
 
-- Python 3.7+
-- [Imagemagick](https://imagemagick.org/script/download.php) for cover extraction from EPUBs (Windows users may need to install [Ghostscript](https://ghostscript.com/releases/gsdnld.html) for PDF cover extraction)
-- Windows users need to install [libmagic for 32bit python](https://gnuwin32.sourceforge.net/downlinks/file.php) or [libmagic for 64bit python](https://github.com/nscaife/file-windows/releases/tag/20170108), depending on the python version; The files need to be installed in path (e.g. script folder of your Calibre-Web venv, or in the root folder of Calibre-Web
-- Optional: [Calibre desktop program](https://calibre-ebook.com/download) for on-the-fly conversion and metadata editing (set "calibre's converter tool" path on the setup page)
-- Optional: [Kepubify tool](https://github.com/pgaskin/kepubify/releases/latest) for Kobo device support (place the binary in `/opt/kepubify` on Linux or `C:\Program Files\kepubify` on Windows)
+- **Python Version**: Ensure you have Python 3.7 or newer.
+- **Imagemagick**: Required for cover extraction from EPUBs. Windows users may also need to install [Ghostscript](https://ghostscript.com/releases/gsdnld.html) for PDF cover extraction.
+- **Windows Users**: Install [libmagic](https://gnuwin32.sourceforge.net/downlinks/file.php) for 32bit python or [libmagic for 64bit python](https://github.com/nscaife/file-windows/releases/tag/20170108), depending on your Python version. Make sure these files are included in your path.
+- **Optional Tools**:
+   - **Calibre desktop program**: Recommended for on-the-fly conversion and metadata editing. Set the path to Calibre’s converter tool on the setup page.
+   - **Kepubify tool**: Needed for Kobo device support. Download the tool and place the binary in `/opt/kepubify` on Linux or `C:\Program Files\kepubify` on Windows.
 
 ## Docker Images
 
-Pre-built Docker images are available in the following Docker Hub repositories (maintained by the LinuxServer team):
+Pre-built Docker images are available:
 
-#### **LinuxServer - x64, aarch64**
-- [Docker Hub](https://hub.docker.com/r/linuxserver/calibre-web)
-- [GitHub](https://github.com/linuxserver/docker-calibre-web)
-- [GitHub - Optional Calibre layer](https://github.com/linuxserver/docker-mods/tree/universal-calibre)
+### **LinuxServer - x64, aarch64**
+- **Docker Hub**: [linuxserver/calibre-web](https://hub.docker.com/r/linuxserver/calibre-web)
+- **GitHub**: [linuxserver/docker-calibre-web](https://github.com/linuxserver/docker-calibre-web)
+- **Optional Calibre layer**: [linuxserver/docker-mods](https://github.com/linuxserver/docker-mods/tree/universal-calibre)
 
-  Include the environment variable `DOCKER_MODS=linuxserver/mods:universal-calibre` in your Docker run/compose file to add the Calibre `ebook-convert` binary (x64 only). Omit this variable for a lightweight image.
+To include the Calibre `ebook-convert` binary (x64 only), add the environment variable:
+``` 
+DOCKER_MODS=linuxserver/mods:universal-calibre
+```
+in your Docker run/compose file. Omit this variable for a lightweight image.
 
-  Both the Calibre-Web and Calibre-Mod images are automatically rebuilt on new releases and updates.
+- **Paths Configuration**:
+   - Set **Path to Calibre Binaries** to `/usr/bin`.
+   - Set **Path to Unrar** to `/usr/bin/unrar`.
 
-  - Set "Path to Calibre Binaries" to `/usr/bin`
-  - Set "Path to Unrar" to `/usr/bin/unrar`
+## Troubleshooting
+
+- **Common Issues**: 
+   - If you experience issues starting the application, check the log files located in the `logs` directory for error messages.
+   - If eBooks fail to load, verify that the `Location of Calibre database` is correctly set and that the database file is accessible.
+
+- **Configuration Errors**: Ensure that your Calibre database is compatible and properly formatted. Refer to the Calibre documentation for guidance on maintaining the database.
+
+- **Performance Problems**: 
+   - If the application is slow, consider increasing the allocated resources (CPU/RAM) to your server or optimizing the Calibre database by removing duplicates and unnecessary entries.
+   - Regularly clear the cache in your web browser to improve loading times.
+
+- **User Management Issues**: If users are unable to log in or register, check the user permission settings in the admin interface. Ensure that registration is enabled and that users are being assigned appropriate roles.
+
+- **Support Resources**: For additional help, consider visiting the [FAQ section](https://github.com/janeczku/calibre-web/wiki/FAQ) of the wiki or posting your questions in the [Discord community](https://discord.gg/h2VsJ2NEfB).
 
 ## Contributor Recognition
 
@@ -123,4 +167,21 @@ For more information, How To's, and FAQs, please visit the [Wiki](https://github
 
 ## Contributing to Calibre-Web
 
-Check out our [Contributing Guidelines](https://github.com/janeczku/calibre-web/blob/master/CONTRIBUTING.md)
+To contribute, please check our [Contributing Guidelines](https://github.com/janeczku/calibre-web/blob/master/CONTRIBUTING.md). We welcome issues, feature requests, and pull requests from the community.
+
+### Reporting Bugs
+
+If you encounter bugs or issues, please report them in the [issues section](https://github.com/janeczku/calibre-web/issues) of the repository. Be sure to include detailed information about your setup and the problem encountered.
+
+### Feature Requests
+
+We welcome suggestions for new features. Please create a new issue in the repository to discuss your ideas.
+
+## Additional Resources
+
+- **Documentation**: Comprehensive documentation is available on the [Calibre-Web wiki](https://github.com/janeczku/calibre-web/wiki).
+- **Community Contributions**: Explore the [community contributions](https://github.com/janeczku/calibre-web/pulls) to see ongoing work and how you can get involved.
+
+---
+
+Thank you for using Calibre-Web! We hope you enjoy managing your eBook library with our tool.

cps/__init__.py

@@ -31,12 +31,10 @@ from flask_principal import Principal
 
 from . import logger
 from .cli import CliParameter
-from .constants import CONFIG_DIR
 from .reverseproxy import ReverseProxied
 from .server import WebServer
 from .dep_check import dependency_check
 from .updater import Updater
-from .babel import babel, get_locale
 from . import config_sql
 from . import cache_buster
 from . import ub, db
@@ -72,6 +70,9 @@ mimetypes.add_type('application/mpeg', '.mpeg')
 mimetypes.add_type('audio/mpeg', '.mp3')
 mimetypes.add_type('audio/x-m4a', '.m4a')
 mimetypes.add_type('audio/x-m4a', '.m4b')
+mimetypes.add_type('audio/x-hx-aac-adts', '.aac')
+mimetypes.add_type('audio/vnd.dolby.dd-raw', '.ac3')
+mimetypes.add_type('video/x-ms-asf', '.asf')
 mimetypes.add_type('audio/ogg', '.ogg')
 mimetypes.add_type('application/ogg', '.oga')
 mimetypes.add_type('text/css', '.css')
@@ -83,9 +84,11 @@ log = logger.create()
 app = Flask(__name__)
 app.config.update(
     SESSION_COOKIE_HTTPONLY=True,
-    SESSION_COOKIE_SAMESITE='Strict',
-    REMEMBER_COOKIE_SAMESITE='Strict',  # will be available in flask-login 0.5.1 earliest
-    WTF_CSRF_SSL_STRICT=False
+    SESSION_COOKIE_SAMESITE='Lax',
+    REMEMBER_COOKIE_SAMESITE='Strict',
+    WTF_CSRF_SSL_STRICT=False,
+    SESSION_COOKIE_NAME=os.environ.get('COOKIE_PREFIX', "") + "session",
+    REMEMBER_COOKIE_NAME=os.environ.get('COOKIE_PREFIX', "") + "remember_token"
 )
 
 lm = MyLoginManager()
@@ -178,6 +181,7 @@ def create_app():
     app.secret_key = os.getenv('SECRET_KEY', config_sql.get_flask_session_key(ub.session))
 
     web_server.init_app(app, config)
+    from .cw_babel import babel, get_locale
     if hasattr(babel, "localeselector"):
         babel.init_app(app)
         babel.localeselector(get_locale)

cps/about.py

@@ -23,10 +23,10 @@
 import sys
 import platform
 import sqlite3
+import importlib
 from collections import OrderedDict
 
 import flask
-import jinja2
 from flask_babel import gettext as _
 
 from . import db, calibre_db, converter, uploader, constants, dep_check
@@ -41,17 +41,18 @@ req = dep_check.load_dependencies(False)
 opt = dep_check.load_dependencies(True)
 for i in (req + opt):
     modules[i[1]] = i[0]
-modules['Jinja2'] = jinja2.__version__
-modules['pySqlite'] = sqlite3.version
+modules['Jinja2'] = importlib.metadata.version("jinja2")
+if sys.version_info < (3, 12):
+    modules['pySqlite'] = sqlite3.version
 modules['SQLite'] = sqlite3.sqlite_version
 sorted_modules = OrderedDict((sorted(modules.items(), key=lambda x: x[0].casefold())))
 
 
 def collect_stats():
     if constants.NIGHTLY_VERSION[0] == "$Format:%H$":
-        calibre_web_version = constants.STABLE_VERSION['version'].replace("b", " Beta")
+        calibre_web_version = constants.STABLE_VERSION.replace("b", " Beta")
     else:
-        calibre_web_version = (constants.STABLE_VERSION['version'].replace("b", " Beta") + ' - '
+        calibre_web_version = (constants.STABLE_VERSION.replace("b", " Beta") + ' - '
                                + constants.NIGHTLY_VERSION[0].replace('%', '%%') + ' - '
                                + constants.NIGHTLY_VERSION[1].replace('%', '%%'))
 

cps/admin.py

@@ -52,8 +52,9 @@ from .gdriveutils import is_gdrive_ready, gdrive_support
 from .render_template import render_title_template, get_sidebar_config
 from .services.worker import WorkerThread
 from .usermanagement import user_login_required
-from .babel import get_available_translations, get_available_locale, get_user_locale_language
+from .cw_babel import get_available_translations, get_available_locale, get_user_locale_language
 from . import debug_info
+from .string_helper import strip_whitespaces
 
 log = logger.create()
 
@@ -305,7 +306,15 @@ def edit_user_table():
         .group_by(text('books_tags_link.tag')) \
         .order_by(db.Tags.name).all()
     if config.config_restricted_column:
-        custom_values = calibre_db.session.query(db.cc_classes[config.config_restricted_column]).all()
+        try:
+            custom_values = calibre_db.session.query(db.cc_classes[config.config_restricted_column]).all()
+        except (KeyError, AttributeError, IndexError):
+            custom_values = []
+            log.error("Custom Column No.{} does not exist in calibre database".format(
+                config.config_restricted_column))
+            flash(_("Custom Column No.%(column)d does not exist in calibre database",
+                    column=config.config_restricted_column),
+                  category="error")
     else:
         custom_values = []
     if not config.config_anonbrowse:
@@ -463,9 +472,9 @@ def edit_list_user(param):
                 if 'value[]' in vals:
                     setattr(user, param, prepare_tags(user, vals['action'][0], param, vals['value[]']))
                 else:
-                    setattr(user, param, vals['value'].strip())
+                    setattr(user, param, strip_whitespaces(vals['value']))
             else:
-                vals['value'] = vals['value'].strip()
+                vals['value'] = strip_whitespaces(vals['value'])
                 if param == 'name':
                     if user.name == "Guest":
                         raise Exception(_("Guest Name can't be changed"))
@@ -566,7 +575,7 @@ def update_view_configuration():
     _config_string(to_save, "config_calibre_web_title")
     _config_string(to_save, "config_columns_to_ignore")
     if _config_string(to_save, "config_title_regex"):
-        calibre_db.update_title_sort(config)
+        calibre_db.create_functions(config)
 
     if not check_valid_read_column(to_save.get("config_read_column", "0")):
         flash(_("Invalid Read Column"), category="error")
@@ -690,7 +699,7 @@ def delete_domain():
 def list_domain(allow):
     answer = ub.session.query(ub.Registration).filter(ub.Registration.allow == allow).all()
     json_dumps = json.dumps([{"domain": r.domain.replace('%', '*').replace('_', '?'), "id": r.id} for r in answer])
-    js = json.dumps(json_dumps.replace('"', "'")).lstrip('"').strip('"')
+    js = json.dumps(json_dumps.replace('"', "'")).strip('"')
     response = make_response(js.replace("'", '"'))
     response.headers["Content-Type"] = "application/json; charset=utf-8"
     return response
@@ -981,8 +990,14 @@ def prepare_tags(user, action, tags_name, id_list):
             raise Exception(_("Tag not found"))
         new_tags_list = [x.name for x in tags]
     else:
-        tags = calibre_db.session.query(db.cc_classes[config.config_restricted_column]) \
-            .filter(db.cc_classes[config.config_restricted_column].id.in_(id_list)).all()
+        try:
+            tags = calibre_db.session.query(db.cc_classes[config.config_restricted_column]) \
+                .filter(db.cc_classes[config.config_restricted_column].id.in_(id_list)).all()
+        except (KeyError, AttributeError, IndexError):
+            log.error("Custom Column No.{} does not exist in calibre database".format(
+                config.config_restricted_column))
+            raise Exception(_("Custom Column No.%(column)d does not exist in calibre database",
+                    column=config.config_restricted_column))
         new_tags_list = [x.value for x in tags]
     saved_tags_list = user.__dict__[tags_name].split(",") if len(user.__dict__[tags_name]) else []
     if action == "remove":
@@ -1100,7 +1115,7 @@ def _config_checkbox_int(to_save, x):
 
 
 def _config_string(to_save, x):
-    return config.set_from_dictionary(to_save, x, lambda y: y.strip().strip(u'\u200B\u200C\u200D\ufeff') if y else y)
+    return config.set_from_dictionary(to_save, x, lambda y: strip_whitespaces(y) if y else y)
 
 
 def _configuration_gdrive_helper(to_save):
@@ -1311,9 +1326,9 @@ def update_mailsettings():
         if to_save.get("mail_password_e", ""):
             _config_string(to_save, "mail_password_e")
         _config_int(to_save, "mail_size", lambda y: int(y) * 1024 * 1024)
-        config.mail_server = to_save.get('mail_server', "").strip()
-        config.mail_from = to_save.get('mail_from', "").strip()
-        config.mail_login = to_save.get('mail_login', "").strip()
+        config.mail_server = strip_whitespaces(to_save.get('mail_server', ""))
+        config.mail_from = strip_whitespaces(to_save.get('mail_from', ""))
+        config.mail_login = strip_whitespaces(to_save.get('mail_login', ""))
     try:
         config.save()
     except (OperationalError, InvalidRequestError) as e:
@@ -1678,10 +1693,10 @@ def cancel_task():
 def _db_simulate_change():
     param = request.form.to_dict()
     to_save = dict()
-    to_save['config_calibre_dir'] = re.sub(r'[\\/]metadata\.db$',
+    to_save['config_calibre_dir'] = strip_whitespaces(re.sub(r'[\\/]metadata\.db$',
                                            '',
                                            param['config_calibre_dir'],
-                                           flags=re.IGNORECASE).strip()
+                                           flags=re.IGNORECASE))
     db_valid, db_change = calibre_db.check_valid_db(to_save["config_calibre_dir"],
                                                     ub.app_DB_path,
                                                     config.config_calibre_uuid)
@@ -1715,6 +1730,13 @@ def _db_configuration_update_helper():
             db_change = True
     except Exception as ex:
         return _db_configuration_result('{}'.format(ex), gdrive_error)
+    config.config_calibre_split = to_save.get('config_calibre_split', 0) == "on"
+    if config.config_calibre_split:
+        split_dir = to_save.get("config_calibre_split_dir")
+        if not os.path.exists(split_dir):
+            return _db_configuration_result(_("Books path not valid"), gdrive_error)
+        else:
+            _config_string(to_save, "config_calibre_split_dir")
 
     if db_change or not db_valid or not config.db_configured \
       or config.config_calibre_dir != to_save["config_calibre_dir"]:
@@ -1736,12 +1758,18 @@ def _db_configuration_update_helper():
             ub.session.query(ub.KoboSyncedBooks).delete()
             helper.delete_thumbnail_cache()
             ub.session_commit()
+            # deleted visibilities based on custom column and tags
+            config.config_restricted_column = 0
+            config.config_denied_tags = ""
+            config.config_allowed_tags = ""
+            config.config_columns_to_ignore = ""
+            config.config_denied_column_value = ""
+            config.config_allowed_column_value = ""
+            config.config_read_column = 0
         _config_string(to_save, "config_calibre_dir")
         calibre_db.update_config(config)
         if not os.access(os.path.join(config.config_calibre_dir, "metadata.db"), os.W_OK):
             flash(_("DB is not Writeable"), category="warning")
-    _config_string(to_save, "config_calibre_split_dir")
-    config.config_calibre_split = to_save.get('config_calibre_split', 0) == "on"
     calibre_db.update_config(config)
     config.save()
     return _db_configuration_result(None, gdrive_error)
@@ -1775,9 +1803,8 @@ def _configuration_update_helper():
 
         if "config_upload_formats" in to_save:
             to_save["config_upload_formats"] = ','.join(
-                helper.uniq([x.lstrip().rstrip().lower() for x in to_save["config_upload_formats"].split(',')]))
+                helper.uniq([x.strip().lower() for x in to_save["config_upload_formats"].split(',')]))
             _config_string(to_save, "config_upload_formats")
-            # constants.EXTENSIONS_UPLOAD = config.config_upload_formats.split(',')
 
         _config_string(to_save, "config_calibre")
         _config_string(to_save, "config_binariesdir")
@@ -2076,7 +2103,7 @@ def _handle_edit_user(to_save, content, languages, translations, kobo_support):
 
 
 def extract_user_data_from_field(user, field):
-    match = re.search(field + r"=([@\.\d\s\w-]+)", user, re.IGNORECASE | re.UNICODE)
+    match = re.search(field + r"=(.*?)($|(?<!\\),)", user, re.IGNORECASE | re.UNICODE)    
     if match:
         return match.group(1)
     else:
@@ -2084,7 +2111,7 @@ def extract_user_data_from_field(user, field):
 
 
 def extract_dynamic_field_from_filter(user, filtr):
-    match = re.search("([a-zA-Z0-9-]+)=%s", filtr, re.IGNORECASE | re.UNICODE)
+    match = re.search(r"([a-zA-Z0-9-]+)=%s", filtr, re.IGNORECASE | re.UNICODE)
     if match:
         return match.group(1)
     else:

cps/audio.py

@@ -0,0 +1,153 @@
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2024 Ozzieisaacs
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import os
+
+import mutagen
+import base64
+from . import cover, logger
+
+from cps.constants import BookMeta
+
+log = logger.create()
+
+def get_audio_file_info(tmp_file_path, original_file_extension, original_file_name, no_cover_processing):
+    tmp_cover_name = None
+    audio_file = mutagen.File(tmp_file_path)
+    comments = None
+    if original_file_extension in [".mp3", ".wav", ".aiff"]:
+        cover_data = list()
+        for key, val in audio_file.tags.items():
+            if key.startswith("APIC:"):
+                cover_data.append(val)
+            if key.startswith("COMM:"):
+                comments = val.text[0]
+        title = audio_file.tags.get('TIT2').text[0] if "TIT2" in audio_file.tags else None
+        author = audio_file.tags.get('TPE1').text[0] if "TPE1" in audio_file.tags else None
+        if author is None:
+            author = audio_file.tags.get('TPE2').text[0] if "TPE2" in audio_file.tags else None
+        tags = audio_file.tags.get('TCON').text[0] if "TCON" in audio_file.tags else None # Genre
+        series = audio_file.tags.get('TALB').text[0] if "TALB" in audio_file.tags else None# Album
+        series_id = audio_file.tags.get('TRCK').text[0] if "TRCK" in audio_file.tags else None # track no.
+        publisher = audio_file.tags.get('TPUB').text[0] if "TPUB" in audio_file.tags else None
+        pubdate = str(audio_file.tags.get('TDRL').text[0]) if "TDRL" in audio_file.tags else None
+        if not pubdate:
+            pubdate = str(audio_file.tags.get('TDRC').text[0]) if "TDRC" in audio_file.tags else None
+            if not pubdate:
+                pubdate = str(audio_file.tags.get('TDOR').text[0]) if "TDOR" in audio_file.tags else None
+        if cover_data and not no_cover_processing:
+            tmp_cover_name = os.path.join(os.path.dirname(tmp_file_path), 'cover.jpg')
+            cover_info = cover_data[0]
+            for dat in cover_data:
+                if dat.type == mutagen.id3.PictureType.COVER_FRONT:
+                    cover_info = dat
+                    break
+            cover.cover_processing(tmp_file_path, cover_info.data, "." + cover_info.mime[-3:])
+    elif original_file_extension in [".ogg", ".flac", ".opus", ".ogv"]:
+        title = audio_file.tags.get('TITLE')[0] if "TITLE" in audio_file else None
+        author = audio_file.tags.get('ARTIST')[0] if "ARTIST" in audio_file else None
+        comments = audio_file.tags.get('COMMENTS')[0] if "COMMENTS" in audio_file else None
+        tags = audio_file.tags.get('GENRE')[0] if "GENRE" in audio_file else None # Genre
+        series = audio_file.tags.get('ALBUM')[0] if "ALBUM" in audio_file else None
+        series_id = audio_file.tags.get('TRACKNUMBER')[0] if "TRACKNUMBER" in audio_file else None
+        publisher = audio_file.tags.get('LABEL')[0] if "LABEL" in audio_file else None
+        pubdate = audio_file.tags.get('DATE')[0] if "DATE" in audio_file else None
+        cover_data = audio_file.tags.get('METADATA_BLOCK_PICTURE')
+        if not no_cover_processing:
+            if cover_data:
+                tmp_cover_name = os.path.join(os.path.dirname(tmp_file_path), 'cover.jpg')
+                cover_info = mutagen.flac.Picture(base64.b64decode(cover_data[0]))
+                cover.cover_processing(tmp_file_path, cover_info.data, "." + cover_info.mime[-3:])
+            if hasattr(audio_file, "pictures"):
+                cover_info = audio_file.pictures[0]
+                for dat in audio_file.pictures:
+                    if dat.type == mutagen.id3.PictureType.COVER_FRONT:
+                        cover_info = dat
+                        break
+                tmp_cover_name = os.path.join(os.path.dirname(tmp_file_path), 'cover.jpg')
+                cover.cover_processing(tmp_file_path, cover_info.data, "." + cover_info.mime[-3:])
+    elif original_file_extension in [".aac"]:
+        title = audio_file.tags.get('Title').value if "Title" in audio_file else None
+        author = audio_file.tags.get('Artist').value if "Artist" in audio_file else None
+        comments = audio_file.tags.get('Comment').value if "Comment" in audio_file else None
+        tags = audio_file.tags.get('Genre').value if "Genre" in audio_file else None
+        series = audio_file.tags.get('Album').value if "Album" in audio_file else None
+        series_id = audio_file.tags.get('Track').value if "Track" in audio_file else None
+        publisher = audio_file.tags.get('Label').value if "Label" in audio_file else None
+        pubdate = audio_file.tags.get('Year').value if "Year" in audio_file else None
+        cover_data = audio_file.tags['Cover Art (Front)']
+        if cover_data and not no_cover_processing:
+            tmp_cover_name = os.path.join(os.path.dirname(tmp_file_path), 'cover.jpg')
+            with open(tmp_cover_name, "wb") as cover_file:
+                cover_file.write(cover_data.value.split(b"\x00",1)[1])
+    elif original_file_extension in [".asf"]:
+        title = audio_file.tags.get('Title')[0].value if "Title" in audio_file else None
+        author = audio_file.tags.get('Artist')[0].value if "Artist" in audio_file else None
+        comments = audio_file.tags.get('Comments')[0].value if "Comments" in audio_file else None
+        tags = audio_file.tags.get('Genre')[0].value if "Genre" in audio_file else None
+        series = audio_file.tags.get('Album')[0].value if "Album" in audio_file else None
+        series_id = audio_file.tags.get('Track')[0].value if "Track" in audio_file else None
+        publisher = audio_file.tags.get('Label')[0].value if "Label" in audio_file else None
+        pubdate = audio_file.tags.get('Year')[0].value if "Year" in audio_file else None
+        cover_data = audio_file.tags.get('WM/Picture', None)
+        if cover_data and not no_cover_processing:
+            tmp_cover_name = os.path.join(os.path.dirname(tmp_file_path), 'cover.jpg')
+            with open(tmp_cover_name, "wb") as cover_file:
+                cover_file.write(cover_data[0].value)
+    elif original_file_extension in [".mp4", ".m4a", ".m4b"]:
+        title = audio_file.tags.get('©nam')[0] if "©nam" in audio_file.tags else None
+        author = audio_file.tags.get('©ART')[0] if "©ART" in audio_file.tags else None
+        comments = audio_file.tags.get('©cmt')[0] if "©cmt" in audio_file.tags else None
+        tags = audio_file.tags.get('©gen')[0] if "©gen" in audio_file.tags else None
+        series = audio_file.tags.get('©alb')[0] if "©alb" in audio_file.tags else None
+        series_id = str(audio_file.tags.get('trkn')[0][0]) if "trkn" in audio_file.tags else None
+        publisher = ""
+        pubdate = audio_file.tags.get('©day')[0] if "©day" in audio_file.tags else None
+        cover_data = audio_file.tags.get('covr', None)
+        if cover_data and not no_cover_processing:
+            tmp_cover_name = os.path.join(os.path.dirname(tmp_file_path), 'cover.jpg')
+            cover_type = None
+            for c in cover_data:
+                if c.imageformat == mutagen.mp4.AtomDataType.JPEG:
+                    cover_type =".jpg"
+                    cover_bin = c
+                    break
+                elif c.imageformat == mutagen.mp4.AtomDataType.PNG:
+                    cover_type = ".png"
+                    cover_bin = c
+                    break
+            if cover_type:
+                cover.cover_processing(tmp_file_path, cover_bin, cover_type)
+            else:
+                logger.error("Unknown covertype in file {} ".format(original_file_name))
+
+    return BookMeta(
+        file_path=tmp_file_path,
+        extension=original_file_extension,
+        title=title or original_file_name ,
+        author="Unknown" if author is None else author,
+        cover=tmp_cover_name,
+        description="" if comments is None else comments,
+        tags="" if tags is None else tags,
+        series="" if series is None else series,
+        series_id="1" if series_id is None else series_id.split("/")[0],
+        languages="",
+        publisher= "" if publisher is None else publisher,
+        pubdate="" if pubdate is None else pubdate,
+        identifiers=[],
+    )

cps/cli.py

@@ -29,8 +29,8 @@ from .constants import DEFAULT_SETTINGS_FILE, DEFAULT_GDRIVE_FILE
 
 def version_info():
     if _NIGHTLY_VERSION[1].startswith('$Format'):
-        return "Calibre-Web version: %s - unknown git-clone" % _STABLE_VERSION['version'].replace("b", " Beta")
-    return "Calibre-Web version: %s -%s" % (_STABLE_VERSION['version'].replace("b", " Beta"), _NIGHTLY_VERSION[1])
+        return "Calibre-Web version: %s - unknown git-clone" % _STABLE_VERSION.replace("b", " Beta")
+    return "Calibre-Web version: %s -%s" % (_STABLE_VERSION.replace("b", " Beta"), _NIGHTLY_VERSION[1])
 
 
 class CliParameter(object):

cps/comic.py

@@ -130,7 +130,7 @@ def _extract_cover(tmp_file_name, original_file_extension, rar_executable):
     return cover.cover_processing(tmp_file_name, cover_data, extension)
 
 
-def get_comic_info(tmp_file_path, original_file_name, original_file_extension, rar_executable):
+def get_comic_info(tmp_file_path, original_file_name, original_file_extension, rar_executable, no_cover_processing):
     if use_comic_meta:
         try:
             archive = ComicArchive(tmp_file_path, rar_exe_path=rar_executable)
@@ -155,14 +155,17 @@ def get_comic_info(tmp_file_path, original_file_name, original_file_extension, r
 
             lang = loaded_metadata.language or ""
             loaded_metadata.language = isoLanguages.get_lang3(lang)
-
+            if not no_cover_processing:
+                cover_file = _extract_cover(tmp_file_path, original_file_extension, rar_executable)
+            else:
+                cover_file = None
             return BookMeta(
                 file_path=tmp_file_path,
                 extension=original_file_extension,
                 title=loaded_metadata.title or original_file_name,
                 author=" & ".join([credit["person"]
                                    for credit in loaded_metadata.credits if credit["role"] == "Writer"]) or 'Unknown',
-                cover=_extract_cover(tmp_file_path, original_file_extension, rar_executable),
+                cover=cover_file,
                 description=loaded_metadata.comments or "",
                 tags="",
                 series=loaded_metadata.series or "",
@@ -171,13 +174,17 @@ def get_comic_info(tmp_file_path, original_file_name, original_file_extension, r
                 publisher="",
                 pubdate="",
                 identifiers=[])
+    if not no_cover_processing:
+        cover_file = _extract_cover(tmp_file_path, original_file_extension, rar_executable)
+    else:
+        cover_file = None
 
     return BookMeta(
         file_path=tmp_file_path,
         extension=original_file_extension,
         title=original_file_name,
         author='Unknown',
-        cover=_extract_cover(tmp_file_path, original_file_extension, rar_executable),
+        cover=cover_file,
         description="",
         tags="",
         series="",

cps/config_sql.py

@@ -35,7 +35,7 @@ except ImportError:
 
 from . import constants, logger
 from .subproc_wrapper import process_wait
-
+from .string_helper import strip_whitespaces
 
 log = logger.create()
 _Base = declarative_base()
@@ -182,26 +182,6 @@ class _Settings(_Base):
 class ConfigSQL(object):
     # pylint: disable=no-member
     def __init__(self):
-        '''self.config_calibre_uuid = None
-        self.config_calibre_split_dir = None
-        self.dirty = None
-        self.config_logfile = None
-        self.config_upload_formats = None
-        self.mail_gmail_token = None
-        self.mail_server_type = None
-        self.mail_server = None
-        self.config_log_level = None
-        self.config_allowed_column_value = None
-        self.config_denied_column_value = None
-        self.config_allowed_tags = None
-        self.config_denied_tags = None
-        self.config_default_show = None
-        self.config_default_role = None
-        self.config_keyfile = None
-        self.config_certfile = None
-        self.config_rarfile_location = None
-        self.config_kepubifypath = None
-        self.config_binariesdir = None'''
         self.__dict__["dirty"] = list()
 
     def init_config(self, session, secret_key, cli):
@@ -288,19 +268,19 @@ class ConfigSQL(object):
 
     def list_denied_tags(self):
         mct = self.config_denied_tags or ""
-        return [t.strip() for t in mct.split(",")]
+        return [strip_whitespaces(t) for t in mct.split(",")]
 
     def list_allowed_tags(self):
         mct = self.config_allowed_tags or ""
-        return [t.strip() for t in mct.split(",")]
+        return [strip_whitespaces(t) for t in mct.split(",")]
 
     def list_denied_column_values(self):
         mct = self.config_denied_column_value or ""
-        return [t.strip() for t in mct.split(",")]
+        return [strip_whitespaces(t) for t in mct.split(",")]
 
     def list_allowed_column_values(self):
         mct = self.config_allowed_column_value or ""
-        return [t.strip() for t in mct.split(",")]
+        return [strip_whitespaces(t) for t in mct.split(",")]
 
     def get_log_level(self):
         return logger.get_level_name(self.config_log_level)
@@ -372,7 +352,7 @@ class ConfigSQL(object):
             db_file = os.path.join(self.config_calibre_dir, 'metadata.db')
             have_metadata_db = os.path.isfile(db_file)
         self.db_configured = have_metadata_db
-        # constants.EXTENSIONS_UPLOAD = [x.lstrip().rstrip().lower() for x in self.config_upload_formats.split(',')]
+        
         from . import cli_param
         if os.environ.get('FLASK_DEBUG'):
             logfile = logger.setup(logger.LOG_TO_STDOUT, logger.logging.DEBUG)

cps/constants.py

@@ -19,9 +19,6 @@
 import sys
 import os
 from collections import namedtuple
-from sqlalchemy import __version__ as sql_version
-
-sqlalchemy_version2 = ([int(x) for x in sql_version.split('.')] >= [2, 0, 0])
 
 # APP_MODE - production, development, or test
 APP_MODE             = os.environ.get('APP_MODE', 'production')
@@ -175,7 +172,7 @@ BookMeta = namedtuple('BookMeta', 'file_path, extension, title, author, cover, d
                                   'series_id, languages, publisher, pubdate, identifiers')
 
 # python build process likes to have x.y.zbw -> b for beta and w a counting number
-STABLE_VERSION = {'version': '0.6.23'}
+STABLE_VERSION =  '0.6.24'
 
 NIGHTLY_VERSION = dict()
 NIGHTLY_VERSION[0] = '$Format:%H$'
@@ -193,7 +190,7 @@ THUMBNAIL_TYPE_AUTHOR    = 3
 COVER_THUMBNAIL_ORIGINAL = 0
 COVER_THUMBNAIL_SMALL    = 1
 COVER_THUMBNAIL_MEDIUM   = 2
-COVER_THUMBNAIL_LARGE    = 3
+COVER_THUMBNAIL_LARGE    = 4
 
 # clean-up the module namespace
 del sys, os, namedtuple

cps/cw_advocate/__init__.py

@@ -0,0 +1,22 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+
+from .adapters import ValidatingHTTPAdapter
+from .api import *
+from .addrvalidator import AddrValidator
+from .exceptions import UnacceptableAddressException

cps/cw_advocate/adapters.py

@@ -0,0 +1,48 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+from requests.adapters import HTTPAdapter, DEFAULT_POOLBLOCK
+
+from .addrvalidator import AddrValidator
+from .exceptions import ProxyDisabledException
+from .poolmanager import ValidatingPoolManager
+
+
+class ValidatingHTTPAdapter(HTTPAdapter):
+    __attrs__ = HTTPAdapter.__attrs__ + ['_validator']
+
+    def __init__(self, *args, **kwargs):
+        self._validator = kwargs.pop('validator', None)
+        if not self._validator:
+            self._validator = AddrValidator()
+        super().__init__(*args, **kwargs)
+
+    def init_poolmanager(self, connections, maxsize, block=DEFAULT_POOLBLOCK,
+                         **pool_kwargs):
+        self._pool_connections = connections
+        self._pool_maxsize = maxsize
+        self._pool_block = block
+        self.poolmanager = ValidatingPoolManager(
+            num_pools=connections,
+            maxsize=maxsize,
+            block=block,
+            validator=self._validator,
+            **pool_kwargs
+        )
+
+    def proxy_manager_for(self, proxy, **proxy_kwargs):
+        raise ProxyDisabledException("Proxies cannot be used with Advocate")

cps/cw_advocate/addrvalidator.py

@@ -0,0 +1,281 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+import functools
+import fnmatch
+import ipaddress
+import re
+
+try:
+    import netifaces
+    HAVE_NETIFACES = True
+except ImportError:
+    netifaces = None
+    HAVE_NETIFACES = False
+
+from .exceptions import NameserverException, ConfigException
+
+
+def canonicalize_hostname(hostname):
+    """Lowercase and punycodify a hostname"""
+    # We do the lowercasing after IDNA encoding because we only want to
+    # lowercase the *ASCII* chars.
+    # TODO: The differences between IDNA2003 and IDNA2008 might be relevant
+    # to us, but both specs are damn confusing.
+    return str(hostname.encode("idna").lower(), 'utf-8')
+
+
+def determine_local_addresses():
+    """Get all IPs that refer to this machine according to netifaces"""
+    if not HAVE_NETIFACES:
+        raise ConfigException("Tried to determine local addresses, "
+                              "but netifaces module was not importable")
+    ips = []
+    for interface in netifaces.interfaces():
+        if_families = netifaces.ifaddresses(interface)
+        for family_kind in {netifaces.AF_INET, netifaces.AF_INET6}:
+            addrs = if_families.get(family_kind, [])
+            for addr in (x.get("addr", "") for x in addrs):
+                if family_kind == netifaces.AF_INET6:
+                    # We can't do anything sensible with the scope here
+                    addr = addr.split("%")[0]
+                ips.append(ipaddress.ip_network(addr))
+    return ips
+
+
+def add_local_address_arg(func):
+    """Add the "_local_addresses" kwarg if it's missing
+
+    IMO this information shouldn't be cached between calls (what if one of the
+    adapters got a new IP at runtime?,) and we don't want each function to
+    recalculate it. Just recalculate it if the caller didn't provide it for us.
+    """
+    @functools.wraps(func)
+    def wrapper(self, *args, **kwargs):
+        if "_local_addresses" not in kwargs:
+            if self.autodetect_local_addresses:
+                kwargs["_local_addresses"] = determine_local_addresses()
+            else:
+                kwargs["_local_addresses"] = []
+        return func(self, *args, **kwargs)
+    return wrapper
+
+
+class AddrValidator:
+    _6TO4_RELAY_NET = ipaddress.ip_network("192.88.99.0/24")
+    # Just the well known prefix, DNS64 servers can set their own
+    # prefix, but in practice most probably don't.
+    _DNS64_WK_PREFIX = ipaddress.ip_network("64:ff9b::/96")
+    DEFAULT_PORT_WHITELIST = {80, 8080, 443, 8443, 8000}
+
+    def __init__(
+            self,
+            ip_blacklist=None,
+            ip_whitelist=None,
+            port_whitelist=None,
+            port_blacklist=None,
+            hostname_blacklist=None,
+            allow_ipv6=False,
+            allow_teredo=False,
+            allow_6to4=False,
+            allow_dns64=False,
+            # Must be explicitly set to "False" if you don't want to try
+            # detecting local interface addresses with netifaces.
+            autodetect_local_addresses=True,
+    ):
+        if not port_blacklist and not port_whitelist:
+            # An assortment of common HTTPS? ports.
+            port_whitelist = self.DEFAULT_PORT_WHITELIST.copy()
+        self.ip_blacklist = ip_blacklist or set()
+        self.ip_whitelist = ip_whitelist or set()
+        self.port_blacklist = port_blacklist or set()
+        self.port_whitelist = port_whitelist or set()
+        # TODO: ATM this can contain either regexes or globs that are converted
+        # to regexes upon every check. Create a collection that automagically
+        # converts them to regexes on insert?
+        self.hostname_blacklist = hostname_blacklist or set()
+        self.allow_ipv6 = allow_ipv6
+        self.allow_teredo = allow_teredo
+        self.allow_6to4 = allow_6to4
+        self.allow_dns64 = allow_dns64
+        self.autodetect_local_addresses = autodetect_local_addresses
+
+    @add_local_address_arg
+    def is_ip_allowed(self, addr_ip, _local_addresses=None):
+        if not isinstance(addr_ip,
+                          (ipaddress.IPv4Address, ipaddress.IPv6Address)):
+            addr_ip = ipaddress.ip_address(addr_ip)
+
+        # The whitelist should take precedence over the blacklist so we can
+        # punch holes in blacklisted ranges
+        if any(addr_ip in net for net in self.ip_whitelist):
+            return True
+
+        if any(addr_ip in net for net in self.ip_blacklist):
+            return False
+
+        if any(addr_ip in net for net in _local_addresses):
+            return False
+
+        if addr_ip.version == 4:
+            if not addr_ip.is_private:
+                # IPs for carrier-grade NAT. Seems weird that it doesn't set
+                # `is_private`, but we need to check `not is_global`
+                if not ipaddress.ip_network(addr_ip).is_global:
+                    return False
+        elif addr_ip.version == 6:
+            # You'd better have a good reason for enabling IPv6
+            # because Advocate's techniques don't work well without NAT.
+            if not self.allow_ipv6:
+                return False
+
+            # v6 addresses can also map to IPv4 addresses! Tricky!
+            v4_nested = []
+            if addr_ip.ipv4_mapped:
+                v4_nested.append(addr_ip.ipv4_mapped)
+            # WTF IPv6? Why you gotta have a billion tunneling mechanisms?
+            # XXX: Do we even really care about these? If we're tunneling
+            # through public servers we shouldn't be able to access
+            # addresses on our private network, right?
+            if addr_ip.sixtofour:
+                if not self.allow_6to4:
+                    return False
+                v4_nested.append(addr_ip.sixtofour)
+            if addr_ip.teredo:
+                if not self.allow_teredo:
+                    return False
+                # Check both the client *and* server IPs
+                v4_nested.extend(addr_ip.teredo)
+            if addr_ip in self._DNS64_WK_PREFIX:
+                if not self.allow_dns64:
+                    return False
+                # When using the well-known prefix the last 4 bytes
+                # are the IPv4 addr
+                v4_nested.append(ipaddress.ip_address(addr_ip.packed[-4:]))
+
+            if not all(self.is_ip_allowed(addr_v4) for addr_v4 in v4_nested):
+                return False
+
+            # fec0::*, apparently deprecated?
+            if addr_ip.is_site_local:
+                return False
+        else:
+            raise ValueError("Unsupported IP version(?): %r" % addr_ip)
+
+        # 169.254.XXX.XXX, AWS uses these for autoconfiguration
+        if addr_ip.is_link_local:
+            return False
+        # 127.0.0.1, ::1, etc.
+        if addr_ip.is_loopback:
+            return False
+        if addr_ip.is_multicast:
+            return False
+        # 192.168.XXX.XXX, 10.XXX.XXX.XXX
+        if addr_ip.is_private:
+            return False
+        # 255.255.255.255, ::ffff:XXXX:XXXX (v6->v4) mapping
+        if addr_ip.is_reserved:
+            return False
+        # There's no reason to connect directly to a 6to4 relay
+        if addr_ip in self._6TO4_RELAY_NET:
+            return False
+        # 0.0.0.0
+        if addr_ip.is_unspecified:
+            return False
+
+        # It doesn't look bad, so... it's must be ok!
+        return True
+
+    def _hostname_matches_pattern(self, hostname, pattern):
+        # If they specified a string, just assume they only want basic globbing.
+        # This stops people from not realizing they're dealing in REs and
+        # not escaping their periods unless they specifically pass in an RE.
+        # This has the added benefit of letting us sanely handle globbed
+        # IDNs by default.
+        if isinstance(pattern, str):
+            # convert the glob to a punycode glob, then a regex
+            pattern = fnmatch.translate(canonicalize_hostname(pattern))
+
+        hostname = canonicalize_hostname(hostname)
+        # Down the line the hostname may get treated as a null-terminated string
+        # (as with `socket.getaddrinfo`.) Try to account for that.
+        #
+        #    >>> socket.getaddrinfo("example.com\x00aaaa", 80)
+        #    [(2, 1, 6, '', ('93.184.216.34', 80)), [...]
+        no_null_hostname = hostname.split("\x00")[0]
+
+        return any(re.match(pattern, x.strip(".")) for x
+                   in (no_null_hostname, hostname))
+
+    def is_hostname_allowed(self, hostname):
+        # Sometimes (like with "external" services that your IP has privileged
+        # access to) you might not always know the IP range to blacklist access
+        # to, or the `A` record might change without you noticing.
+        # For e.x.: `foocorp.external.org`.
+        #
+        # Another option is doing something like:
+        #
+        #     for addrinfo in socket.getaddrinfo("foocorp.external.org", 80):
+        #         global_validator.ip_blacklist.add(ip_address(addrinfo[4][0]))
+        #
+        # but that's not always a good idea if they're behind a third-party lb.
+        for pattern in self.hostname_blacklist:
+            if self._hostname_matches_pattern(hostname, pattern):
+                return False
+        return True
+
+    @add_local_address_arg
+    def is_addrinfo_allowed(self, addrinfo, _local_addresses=None):
+        assert(len(addrinfo) == 5)
+        # XXX: Do we care about any of the other elements? Guessing not.
+        family, socktype, proto, canonname, sockaddr = addrinfo
+
+        # The 4th elem inaddrinfo may either be a touple of two or four items,
+        # depending on whether we're dealing with IPv4 or v6
+        if len(sockaddr) == 2:
+            # v4
+            ip, port = sockaddr
+        elif len(sockaddr) == 4:
+            # v6
+            # XXX: what *are* `flow_info` and `scope_id`? Anything useful?
+            # Seems like we can figure out all we need about the scope from
+            # the `is_<x>` properties.
+            ip, port, flow_info, scope_id = sockaddr
+        else:
+            raise ValueError("Unexpected addrinfo format %r" % sockaddr)
+
+        # Probably won't help protect against SSRF, but might prevent our being
+        # used to attack others' non-HTTP services. See
+        # http://www.remote.org/jochen/sec/hfpa/
+        if self.port_whitelist and port not in self.port_whitelist:
+            return False
+        if port in self.port_blacklist:
+            return False
+
+        if self.hostname_blacklist:
+            if not canonname:
+                raise NameserverException(
+                    "addrinfo must contain the canon name to do blacklisting "
+                    "based on hostname. Make sure you use the "
+                    "`socket.AI_CANONNAME` flag, and that each record contains "
+                    "the canon name. Your DNS server might also be garbage."
+                )
+
+            if not self.is_hostname_allowed(canonname):
+                return False
+
+        return self.is_ip_allowed(ip, _local_addresses=_local_addresses)

cps/cw_advocate/api.py

@@ -0,0 +1,202 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+"""
+advocate.api
+~~~~~~~~~~~~
+
+This module implements the Requests API, largely a copy/paste from `requests`
+itself.
+
+:copyright: (c) 2015 by Jordan Milne.
+:license: Apache2, see LICENSE for more details.
+
+"""
+from collections import OrderedDict
+import hashlib
+import pickle
+
+from requests import Session as RequestsSession
+
+# import cw_advocate
+from .adapters import ValidatingHTTPAdapter
+from .exceptions import MountDisabledException
+
+
+class Session(RequestsSession):
+    """Convenience wrapper around `requests.Session` set up for `advocate`ing"""
+
+    __attrs__ = RequestsSession.__attrs__ + ["validator"]
+    DEFAULT_VALIDATOR = None
+    """
+    User-replaceable default validator to use for all Advocate sessions,
+    includes sessions created by advocate.get()
+    """
+
+    def __init__(self, *args, **kwargs):
+        self.validator = kwargs.pop("validator", None) or self.DEFAULT_VALIDATOR
+        adapter_kwargs = kwargs.pop("_adapter_kwargs", {})
+
+        # `Session.__init__()` calls `mount()` internally, so we need to allow
+        # it temporarily
+        self.__mount_allowed = True
+        RequestsSession.__init__(self, *args, **kwargs)
+
+        # Drop any existing adapters
+        self.adapters = OrderedDict()
+
+        self.mount("http://", ValidatingHTTPAdapter(validator=self.validator, **adapter_kwargs))
+        self.mount("https://", ValidatingHTTPAdapter(validator=self.validator, **adapter_kwargs))
+        self.__mount_allowed = False
+
+    def mount(self, *args, **kwargs):
+        """Wrapper around `mount()` to prevent a protection bypass"""
+        if self.__mount_allowed:
+            super().mount(*args, **kwargs)
+        else:
+            raise MountDisabledException(
+                "mount() is disabled to prevent protection bypasses"
+            )
+
+
+def session(*args, **kwargs):
+    return Session(*args, **kwargs)
+
+
+def request(method, url, **kwargs):
+    """Constructs and sends a :class:`Request <Request>`.
+
+    :param method: method for the new :class:`Request` object.
+    :param url: URL for the new :class:`Request` object.
+    :param params: (optional) Dictionary or bytes to be sent in the query string for the :class:`Request`.
+    :param data: (optional) Dictionary, bytes, or file-like object to send in the body of the :class:`Request`.
+    :param json: (optional) json data to send in the body of the :class:`Request`.
+    :param headers: (optional) Dictionary of HTTP Headers to send with the :class:`Request`.
+    :param cookies: (optional) Dict or CookieJar object to send with the :class:`Request`.
+    :param files: (optional) Dictionary of ``'name': file-like-objects`` (or ``{'name': ('filename', fileobj)}``) for multipart encoding upload.
+    :param auth: (optional) Auth tuple to enable Basic/Digest/Custom HTTP Auth.
+    :param timeout: (optional) How long to wait for the server to send data
+        before giving up, as a float, or a (`connect timeout, read timeout
+        <user/advanced.html#timeouts>`_) tuple.
+    :type timeout: float or tuple
+    :param allow_redirects: (optional) Boolean. Set to True if POST/PUT/DELETE redirect following is allowed.
+    :type allow_redirects: bool
+    :param proxies: (optional) Dictionary mapping protocol to the URL of the proxy.
+    :param verify: (optional) if ``True``, the SSL cert will be verified. A CA_BUNDLE path can also be provided.
+    :param stream: (optional) if ``False``, the response content will be immediately downloaded.
+    :param cert: (optional) if String, path to ssl client cert file (.pem). If Tuple, ('cert', 'key') pair.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
+    """
+
+    validator = kwargs.pop("validator", None)
+    with Session(validator=validator) as sess:
+        response = sess.request(method=method, url=url, **kwargs)
+    return response
+
+
+def get(url, **kwargs):
+    """Sends a GET request.
+
+    :param url: URL for the new :class:`Request` object.
+    :param **kwargs: Optional arguments that ``request`` takes.
+    :return: :class:`Response <Response>` object
+    :rtype: requests.Response
+    """
+
+    kwargs.setdefault('allow_redirects', True)
+    return request('get', url, **kwargs)
+
+
+class RequestsAPIWrapper:
+    """Provides a `requests.api`-like interface with a specific validator"""
+
+    # Due to how the classes are dynamically constructed pickling may not work
+    # correctly unless loaded within the same interpreter instance.
+    # Enable at your peril.
+    SUPPORT_WRAPPER_PICKLING = False
+
+    def __init__(self, validator):
+        # Do this here to avoid circular import issues
+        try:
+            from .futures import FuturesSession
+            have_requests_futures = True
+        except ImportError as e:
+            have_requests_futures = False
+
+        self.validator = validator
+        outer_self = self
+
+        class _WrappedSession(Session):
+            """An `advocate.Session` that uses the wrapper's blacklist
+
+            the wrapper is meant to be a transparent replacement for `requests`,
+            so people should be able to subclass `wrapper.Session` and still
+            get the desired validation behaviour
+            """
+            DEFAULT_VALIDATOR = outer_self.validator
+
+        self._make_wrapper_cls_global(_WrappedSession)
+
+        if have_requests_futures:
+
+            class _WrappedFuturesSession(FuturesSession):
+                """Like _WrappedSession, but for `FuturesSession`s"""
+                DEFAULT_VALIDATOR = outer_self.validator
+            self._make_wrapper_cls_global(_WrappedFuturesSession)
+
+            self.FuturesSession = _WrappedFuturesSession
+
+        self.request = self._default_arg_wrapper(request)
+        self.get = self._default_arg_wrapper(get)
+        self.Session = _WrappedSession
+
+    def __getattr__(self, item):
+        # This class is meant to mimic the requests base module, so if we don't
+        # have this attribute, it might be on the base module (like the Request
+        # class, etc.)
+        try:
+            return object.__getattribute__(self, item)
+        except AttributeError:
+            from . import cw_advocate
+            return getattr(cw_advocate, item)
+
+    def _default_arg_wrapper(self, fun):
+        def wrapped_func(*args, **kwargs):
+            kwargs.setdefault("validator", self.validator)
+            return fun(*args, **kwargs)
+        return wrapped_func
+
+    def _make_wrapper_cls_global(self, cls):
+        if not self.SUPPORT_WRAPPER_PICKLING:
+            return
+        # Gnarly, but necessary to give pickle a consistent module-level
+        # reference for each wrapper.
+        wrapper_hash = hashlib.sha256(pickle.dumps(self)).hexdigest()
+        cls.__name__ = "_".join((cls.__name__, wrapper_hash))
+        cls.__qualname__ = ".".join((__name__, cls.__name__))
+        if not globals().get(cls.__name__):
+            globals()[cls.__name__] = cls
+
+
+__all__ = (
+    "get",
+    "request",
+    "session",
+    "Session",
+    "RequestsAPIWrapper",
+)

cps/cw_advocate/connection.py

@@ -0,0 +1,201 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+import ipaddress
+import socket
+from socket import timeout as SocketTimeout
+
+from urllib3.connection import HTTPSConnection, HTTPConnection
+from urllib3.exceptions import ConnectTimeoutError
+from urllib3.util.connection import _set_socket_options
+from urllib3.util.connection import create_connection as old_create_connection
+
+from . import addrvalidator
+from .exceptions import UnacceptableAddressException
+
+
+def advocate_getaddrinfo(host, port, get_canonname=False):
+    addrinfo = socket.getaddrinfo(
+        host,
+        port,
+        0,
+        socket.SOCK_STREAM,
+        0,
+        # We need what the DNS client sees the hostname as, correctly handles
+        # IDNs and tricky things like `private.foocorp.org\x00.google.com`.
+        # All IDNs will be converted to punycode.
+        socket.AI_CANONNAME if get_canonname else 0,
+    )
+    return fix_addrinfo(addrinfo)
+
+
+def fix_addrinfo(records):
+    """
+    Propagate the canonname across records and parse IPs
+
+    I'm not sure if this is just the behaviour of `getaddrinfo` on Linux, but
+    it seems like only the first record in the set has the canonname field
+    populated.
+    """
+    def fix_record(record, canonname):
+        sa = record[4]
+        sa = (ipaddress.ip_address(sa[0]),) + sa[1:]
+        return record[0], record[1], record[2], canonname, sa
+
+    canonname = None
+    if records:
+        # Apparently the canonical name is only included in the first record?
+        # Add it to all of them.
+        assert(len(records[0]) == 5)
+        canonname = records[0][3]
+    return tuple(fix_record(x, canonname) for x in records)
+
+
+# Lifted from requests' urllib3, which in turn lifted it from `socket.py`. Oy!
+def validating_create_connection(address,
+                       timeout=socket._GLOBAL_DEFAULT_TIMEOUT,
+                       source_address=None, socket_options=None,
+                       validator=None):
+    """Connect to *address* and return the socket object.
+
+    Convenience function.  Connect to *address* (a 2-tuple ``(host,
+    port)``) and return the socket object.  Passing the optional
+    *timeout* parameter will set the timeout on the socket instance
+    before attempting to connect.  If no *timeout* is supplied, the
+    global default timeout setting returned by :func:`getdefaulttimeout`
+    is used.  If *source_address* is set it must be a tuple of (host, port)
+    for the socket to bind as a source address before making the connection.
+    An host of '' or port 0 tells the OS to use the default.
+    """
+
+    host, port = address
+    # We can skip asking for the canon name if we're not doing hostname-based
+    # blacklisting.
+    need_canonname = False
+    if validator.hostname_blacklist:
+        need_canonname = True
+        # We check both the non-canonical and canonical hostnames so we can
+        # catch both of these:
+        # CNAME from nonblacklisted.com -> blacklisted.com
+        # CNAME from blacklisted.com -> nonblacklisted.com
+        if not validator.is_hostname_allowed(host):
+            raise UnacceptableAddressException(host)
+
+    err = None
+    addrinfo = advocate_getaddrinfo(host, port, get_canonname=need_canonname)
+    if addrinfo:
+        if validator.autodetect_local_addresses:
+            local_addresses = addrvalidator.determine_local_addresses()
+        else:
+            local_addresses = []
+        for res in addrinfo:
+            # Are we allowed to connect with this result?
+            if not validator.is_addrinfo_allowed(
+                res,
+                _local_addresses=local_addresses,
+            ):
+                continue
+            af, socktype, proto, canonname, sa = res
+            # Unparse the validated IP
+            sa = (sa[0].exploded,) + sa[1:]
+            sock = None
+            try:
+                sock = socket.socket(af, socktype, proto)
+
+                # If provided, set socket level options before connecting.
+                # This is the only addition urllib3 makes to this function.
+                _set_socket_options(sock, socket_options)
+
+                if timeout is not socket._GLOBAL_DEFAULT_TIMEOUT:
+                    sock.settimeout(timeout)
+                if source_address:
+                    sock.bind(source_address)
+                sock.connect(sa)
+                return sock
+
+            except socket.error as _:
+                err = _
+                if sock is not None:
+                    sock.close()
+                    sock = None
+
+        if err is None:
+            # If we got here, none of the results were acceptable
+            err = UnacceptableAddressException(address)
+    if err is not None:
+        raise err
+    else:
+        raise socket.error("getaddrinfo returns an empty list")
+
+
+# TODO: Is there a better way to add this to multiple classes with different
+# base classes? I tried a mixin, but it used the base method instead.
+def _validating_new_conn(self):
+    """ Establish a socket connection and set nodelay settings on it.
+
+    :return: New socket connection.
+    """
+    extra_kw = {}
+    if self.source_address:
+        extra_kw['source_address'] = self.source_address
+
+    if self.socket_options:
+        extra_kw['socket_options'] = self.socket_options
+
+    try:
+        # Hack around HTTPretty's patched sockets
+        # TODO: some better method of hacking around it that checks if we
+        # _would have_ connected to a private addr?
+        conn_func = validating_create_connection
+        if socket.getaddrinfo.__module__.startswith("httpretty"):
+            conn_func = old_create_connection
+        else:
+            extra_kw["validator"] = self._validator
+
+        conn = conn_func(
+            (self.host, self.port),
+            self.timeout,
+            **extra_kw
+        )
+
+    except SocketTimeout:
+        raise ConnectTimeoutError(
+            self, "Connection to %s timed out. (connect timeout=%s)" %
+            (self.host, self.timeout))
+
+    return conn
+
+
+# Don't silently break if the private API changes across urllib3 versions
+assert(hasattr(HTTPConnection, '_new_conn'))
+assert(hasattr(HTTPSConnection, '_new_conn'))
+
+
+class ValidatingHTTPConnection(HTTPConnection):
+    _new_conn = _validating_new_conn
+
+    def __init__(self, *args, **kwargs):
+        self._validator = kwargs.pop("validator")
+        HTTPConnection.__init__(self, *args, **kwargs)
+
+
+class ValidatingHTTPSConnection(HTTPSConnection):
+    _new_conn = _validating_new_conn
+
+    def __init__(self, *args, **kwargs):
+        self._validator = kwargs.pop("validator")
+        HTTPSConnection.__init__(self, *args, **kwargs)

cps/cw_advocate/connectionpool.py

@@ -0,0 +1,39 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+from urllib3 import HTTPConnectionPool, HTTPSConnectionPool
+
+from .connection import (
+    ValidatingHTTPConnection,
+    ValidatingHTTPSConnection,
+)
+
+# Don't silently break if the private API changes across urllib3 versions
+assert(hasattr(HTTPConnectionPool, 'ConnectionCls'))
+assert(hasattr(HTTPSConnectionPool, 'ConnectionCls'))
+assert(hasattr(HTTPConnectionPool, 'scheme'))
+assert(hasattr(HTTPSConnectionPool, 'scheme'))
+
+
+class ValidatingHTTPConnectionPool(HTTPConnectionPool):
+    scheme = 'http'
+    ConnectionCls = ValidatingHTTPConnection
+
+
+class ValidatingHTTPSConnectionPool(HTTPSConnectionPool):
+    scheme = 'https'
+    ConnectionCls = ValidatingHTTPSConnection

cps/cw_advocate/exceptions.py

@@ -0,0 +1,39 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+class AdvocateException(Exception):
+    pass
+
+
+class UnacceptableAddressException(AdvocateException):
+    pass
+
+
+class NameserverException(AdvocateException):
+    pass
+
+
+class MountDisabledException(AdvocateException):
+    pass
+
+
+class ProxyDisabledException(NotImplementedError, AdvocateException):
+    pass
+
+
+class ConfigException(AdvocateException):
+    pass

cps/cw_advocate/poolmanager.py

@@ -0,0 +1,61 @@
+#
+# Copyright 2015 Jordan Milne
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# Source: https://github.com/JordanMilne/Advocate
+
+import collections
+import functools
+
+from urllib3 import PoolManager
+from urllib3.poolmanager import _default_key_normalizer, PoolKey
+
+from .connectionpool import (
+    ValidatingHTTPSConnectionPool,
+    ValidatingHTTPConnectionPool,
+)
+
+pool_classes_by_scheme = {
+    "http": ValidatingHTTPConnectionPool,
+    "https": ValidatingHTTPSConnectionPool,
+}
+
+AdvocatePoolKey = collections.namedtuple('AdvocatePoolKey',
+                                         PoolKey._fields + ('key_validator',))
+
+
+def key_normalizer(key_class, request_context):
+    request_context = request_context.copy()
+    # TODO: add ability to serialize validator rules to dict,
+    # allowing pool to be shared between sessions with the same
+    # rules.
+    request_context["validator"] = id(request_context["validator"])
+    return _default_key_normalizer(key_class, request_context)
+
+
+key_fn_by_scheme = {
+    'http': functools.partial(key_normalizer, AdvocatePoolKey),
+    'https': functools.partial(key_normalizer, AdvocatePoolKey),
+}
+
+
+class ValidatingPoolManager(PoolManager):
+    def __init__(self, *args, **kwargs):
+        super().__init__(*args, **kwargs)
+
+        # Make sure the API hasn't changed
+        assert (hasattr(self, 'pool_classes_by_scheme'))
+
+        self.pool_classes_by_scheme = pool_classes_by_scheme
+        self.key_fn_by_scheme = key_fn_by_scheme.copy()

cps/cw_login/login_manager.py

@@ -1,4 +1,5 @@
 from datetime import datetime
+from datetime import timezone
 from datetime import timedelta
 import hashlib
 
@@ -496,7 +497,7 @@ class LoginManager:
             duration = timedelta(seconds=duration)
 
         try:
-            expires = datetime.utcnow() + duration
+            expires = datetime.now(timezone.utc) + duration
         except TypeError as e:
             raise Exception(
                 "REMEMBER_COOKIE_DURATION must be a datetime.timedelta,"

cps/db.py

@@ -20,10 +20,11 @@
 import os
 import re
 import json
-from datetime import datetime
+from datetime import datetime, timezone
 from urllib.parse import quote
 import unidecode
 from weakref import WeakSet
+from uuid import uuid4
 
 from sqlite3 import OperationalError as sqliteOperationalError
 from sqlalchemy import create_engine
@@ -48,7 +49,7 @@ from flask import flash
 
 from . import logger, ub, isoLanguages
 from .pagination import Pagination
-
+from .string_helper import strip_whitespaces
 
 log = logger.create()
 
@@ -113,7 +114,7 @@ class Identifiers(Base):
         if format_type == 'amazon':
             return "Amazon"
         elif format_type.startswith("amazon_"):
-            return "Amazon.{0}".format(format_type[7:])
+            return "Amazon.{0}".format(format_type[7:].lower().replace("uk","co.uk"))
         elif format_type == "isbn":
             return "ISBN"
         elif format_type == "doi":
@@ -148,7 +149,7 @@ class Identifiers(Base):
         if format_type == "amazon" or format_type == "asin":
             return "https://amazon.com/dp/{0}".format(self.val)
         elif format_type.startswith('amazon_'):
-            return "https://amazon.{0}/dp/{1}".format(format_type[7:], self.val)
+            return "https://amazon.{0}/dp/{1}".format(format_type[7:].lower().replace("uk","co.uk"), self.val)
         elif format_type == "isbn":
             return "https://www.worldcat.org/isbn/{0}".format(self.val)
         elif format_type == "doi":
@@ -378,10 +379,10 @@ class Books(Base):
     title = Column(String(collation='NOCASE'), nullable=False, default='Unknown')
     sort = Column(String(collation='NOCASE'))
     author_sort = Column(String(collation='NOCASE'))
-    timestamp = Column(TIMESTAMP, default=datetime.utcnow)
+    timestamp = Column(TIMESTAMP, default=lambda: datetime.now(timezone.utc))
     pubdate = Column(TIMESTAMP, default=DEFAULT_PUBDATE)
     series_index = Column(String, nullable=False, default="1.0")
-    last_modified = Column(TIMESTAMP, default=datetime.utcnow)
+    last_modified = Column(TIMESTAMP, default=lambda: datetime.now(timezone.utc))
     path = Column(String, default="", nullable=False)
     has_cover = Column(Integer, default=0)
     uuid = Column(String)
@@ -533,7 +534,7 @@ class CalibreDB:
     def init_session(self, expire_on_commit=True):
         self.session = self.session_factory()
         self.session.expire_on_commit = expire_on_commit
-        self.update_title_sort(self.config)
+        self.create_functions(self.config)
 
     @classmethod
     def setup_db_cc_classes(cls, cc):
@@ -875,10 +876,11 @@ class CalibreDB:
             authors_ordered = list()
             # error = False
             for auth in sort_authors:
-                results = self.session.query(Authors).filter(Authors.sort == auth.lstrip().strip()).all()
+                auth = strip_whitespaces(auth)
+                results = self.session.query(Authors).filter(Authors.sort == auth).all()
                 # ToDo: How to handle not found author name
                 if not len(results):
-                    log.error("Author {} not found to display name in right order".format(auth.strip()))
+                    log.error("Author {} not found to display name in right order".format(auth))
                     # error = True
                     break
                 for r in results:
@@ -900,7 +902,8 @@ class CalibreDB:
 
     def get_typeahead(self, database, query, replace=('', ''), tag_filter=true()):
         query = query or ''
-        self.session.connection().connection.connection.create_function("lower", 1, lcase)
+        self.create_functions()
+        # self.session.connection().connection.connection.create_function("lower", 1, lcase)
         entries = self.session.query(database).filter(tag_filter). \
             filter(func.lower(database.name).ilike("%" + query + "%")).all()
         # json_dumps = json.dumps([dict(name=escape(r.name.replace(*replace))) for r in entries])
@@ -908,7 +911,8 @@ class CalibreDB:
         return json_dumps
 
     def check_exists_book(self, authr, title):
-        self.session.connection().connection.connection.create_function("lower", 1, lcase)
+        self.create_functions()
+        # self.session.connection().connection.connection.create_function("lower", 1, lcase)
         q = list()
         author_terms = re.split(r'\s*&\s*', authr)
         for author_term in author_terms:
@@ -918,8 +922,9 @@ class CalibreDB:
             .filter(and_(Books.authors.any(and_(*q)), func.lower(Books.title).ilike("%" + title + "%"))).first()
 
     def search_query(self, term, config, *join):
-        term.strip().lower()
-        self.session.connection().connection.connection.create_function("lower", 1, lcase)
+        strip_whitespaces(term).lower()
+        self.create_functions()
+        # self.session.connection().connection.connection.create_function("lower", 1, lcase)
         q = list()
         author_terms = re.split("[, ]+", term)
         for author_term in author_terms:
@@ -1017,7 +1022,7 @@ class CalibreDB:
                 lang.name = isoLanguages.get_language_name(get_locale(), lang.lang_code)
             return sorted(languages, key=lambda x: x.name, reverse=reverse_order)
 
-    def update_title_sort(self, config, conn=None):
+    def create_functions(self, config=None):
         # user defined sort function for calibre databases (Series, etc.)
         def _title_sort(title):
             # calibre sort stuff
@@ -1026,16 +1031,19 @@ class CalibreDB:
             if match:
                 prep = match.group(1)
                 title = title[len(prep):] + ', ' + prep
-            return title.strip()
+            return strip_whitespaces(title)
 
         try:
-            # sqlalchemy <1.4.24
-            conn = conn or self.session.connection().connection.driver_connection
+            # sqlalchemy <1.4.24 and sqlalchemy 2.0
+            conn = self.session.connection().connection.driver_connection
         except AttributeError:
-            # sqlalchemy >1.4.24 and sqlalchemy 2.0
-            conn = conn or self.session.connection().connection.connection
+            # sqlalchemy >1.4.24
+            conn = self.session.connection().connection.connection
         try:
-            conn.create_function("title_sort", 1, _title_sort)
+            if config:
+                conn.create_function("title_sort", 1, _title_sort)
+            conn.create_function('uuid4', 0, lambda: str(uuid4()))
+            conn.create_function("lower", 1, lcase)
         except sqliteOperationalError:
             pass
 

cps/debug_info.py

@@ -26,7 +26,8 @@ from flask_babel.speaklater import LazyString
 
 import os
 
-from flask import send_file, __version__
+from flask import send_file
+import importlib
 
 from . import logger, config
 from .about import collect_stats
@@ -49,7 +50,8 @@ def assemble_logfiles(file_name):
         with open(f, 'rb') as fd:
             shutil.copyfileobj(fd, wfd)
     wfd.seek(0)
-    if int(__version__.split('.')[0]) < 2:
+    version = importlib.metadata.version("flask")
+    if int(version.split('.')[0]) < 2:
         return send_file(wfd,
                          as_attachment=True,
                          attachment_filename=os.path.basename(file_name))
@@ -72,7 +74,8 @@ def send_debug():
         for fp in file_list:
             zf.write(fp, os.path.basename(fp))
     memory_zip.seek(0)
-    if int(__version__.split('.')[0]) < 2:
+    version = importlib.metadata.version("flask")
+    if int(version.split('.')[0]) < 2:
         return send_file(memory_zip,
                          as_attachment=True,
                          attachment_filename="Calibre-Web-debug-pack.zip")

cps/dep_check.py

@@ -39,11 +39,27 @@ def load_dependencies(optional=False):
             with open(req_path, 'r') as f:
                 for line in f:
                     if not line.startswith('#') and not line == '\n' and not line.startswith('git'):
-                        res = re.match(r'(.*?)([<=>\s]+)([\d\.]+),?\s?([<=>\s]+)?([\d\.]+)?', line.strip())
+                        res = re.match(r'(.*?)([<=>\s]+)([\d\.]+),?\s?([<=>\s]+)?([\d\.]+)?(?:\s?;\s?'
+                                       r'(?:(python_version)\s?([<=>]+)\s?\'([\d\.]+)\'|'
+                                       r'(sys_platform)\s?([\!=]+)\s?\'([\w]+)\'))?', line.strip())
                         try:
                             if getattr(sys, 'frozen', False):
                                 dep_version = exe_deps[res.group(1).lower().replace('_', '-')]
                             else:
+                                if res.group(7) and res.group(8):
+                                    val = res.group(8).split(".")
+                                    if not eval(str(sys.version_info[0]) + "." + "{:02d}".format(sys.version_info[1]) +
+                                                res.group(7) + val[0] + "." + "{:02d}".format(int(val[1]))):
+                                        continue
+                                elif res.group(10) and res.group(11):
+                                    # only installed if platform is eqal, don't check if platform is not equal
+                                    if res.group(10) == "==":
+                                        if sys.platform != res.group(11):
+                                            continue
+                                    # installed if platform is not eqal, don't check if platform is equal
+                                    elif res.group(10) == "!=":
+                                        if sys.platform == res.group(11):
+                                            continue
                                 if importlib:
                                     dep_version = version(res.group(1))
                                 else:
@@ -63,7 +79,7 @@ def dependency_check(optional=False):
     deps = load_dependencies(optional)
     for dep in deps:
         try:
-            dep_version_int = [int(x) if x.isnumeric() else 0 for x in dep[0].split('.')]
+            dep_version_int = [int(x) if x.isnumeric() else 0 for x in dep[0].split('.')[:3]]
             low_check = [int(x) for x in dep[3].split('.')]
             high_check = [int(x) for x in dep[5].split('.')]
         except AttributeError:

cps/editbooks.py

@@ -21,10 +21,10 @@
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
 import os
-from datetime import datetime
+from datetime import datetime, timezone
 import json
 from shutil import copyfile
-from uuid import uuid4
+
 from markupsafe import escape, Markup  # dependency of flask
 from functools import wraps
 
@@ -47,7 +47,7 @@ from .kobo_sync_status import change_archived_books
 from .redirect import get_redirect_location
 from .file_helper import validate_mime_type
 from .usermanagement import user_login_required, login_required_if_no_ano
-
+from .string_helper import strip_whitespaces
 
 editbook = Blueprint('edit-book', __name__)
 log = logger.create()
@@ -97,157 +97,22 @@ def show_edit_book(book_id):
 @login_required_if_no_ano
 @edit_required
 def edit_book(book_id):
-    modify_date = False
-    edit_error = False
-
-    # create the function for sorting...
-    calibre_db.update_title_sort(config)
-
-    book = calibre_db.get_filtered_book(book_id, allow_show_archived=True)
-    # Book not found
-    if not book:
-        flash(_("Oops! Selected book is unavailable. File does not exist or is not accessible"),
-              category="error")
-        return redirect(url_for("web.index"))
-
-    to_save = request.form.to_dict()
-
-    try:
-        # Update folder of book on local disk
-        edited_books_id = None
-        title_author_error = None
-        # handle book title change
-        title_change = handle_title_on_edit(book, to_save["book_title"])
-        # handle book author change
-        input_authors, author_change = handle_author_on_edit(book, to_save["author_name"])
-        if author_change or title_change:
-            edited_books_id = book.id
-            modify_date = True
-            title_author_error = helper.update_dir_structure(edited_books_id,
-                                                             config.get_book_path(),
-                                                             input_authors[0])
-        if title_author_error:
-            flash(title_author_error, category="error")
-            calibre_db.session.rollback()
-            book = calibre_db.get_filtered_book(book_id, allow_show_archived=True)
-
-        # handle upload other formats from local disk
-        meta = upload_single_file(request, book, book_id)
-        # only merge metadata if file was uploaded and no error occurred (meta equals not false or none)
-        upload_format = False
-        if meta:
-            upload_format = merge_metadata(to_save, meta)
-        # handle upload covers from local disk
-        cover_upload_success = upload_cover(request, book)
-        if cover_upload_success:
-            book.has_cover = 1
-            modify_date = True
-
-        # upload new covers or new file formats to google drive
-        if config.config_use_google_drive:
-            gdriveutils.updateGdriveCalibreFromLocal()
-
-        if to_save.get("cover_url", None):
-            if not current_user.role_upload():
-                edit_error = True
-                flash(_("User has no rights to upload cover"), category="error")
-            if to_save["cover_url"].endswith('/static/generic_cover.jpg'):
-                book.has_cover = 0
-            else:
-                result, error = helper.save_cover_from_url(to_save["cover_url"].strip(), book.path)
-                if result is True:
-                    book.has_cover = 1
-                    modify_date = True
-                    helper.replace_cover_thumbnail_cache(book.id)
-                else:
-                    flash(error, category="error")
-
-        # Add default series_index to book
-        modify_date |= edit_book_series_index(to_save["series_index"], book)
-        # Handle book comments/description
-        modify_date |= edit_book_comments(Markup(to_save['description']).unescape(), book)
-        # Handle identifiers
-        input_identifiers = identifier_list(to_save, book)
-        modification, warning = modify_identifiers(input_identifiers, book.identifiers, calibre_db.session)
-        if warning:
-            flash(_("Identifiers are not Case Sensitive, Overwriting Old Identifier"), category="warning")
-        modify_date |= modification
-        # Handle book tags
-        modify_date |= edit_book_tags(to_save['tags'], book)
-        # Handle book series
-        modify_date |= edit_book_series(to_save["series"], book)
-        # handle book publisher
-        modify_date |= edit_book_publisher(to_save['publisher'], book)
-        # handle book languages
-        try:
-            modify_date |= edit_book_languages(to_save['languages'], book, upload_format)
-        except ValueError as e:
-            flash(str(e), category="error")
-            edit_error = True
-        # handle book ratings
-        modify_date |= edit_book_ratings(to_save, book)
-        # handle cc data
-        modify_date |= edit_all_cc_data(book_id, book, to_save)
-
-        if to_save.get("pubdate", None):
-            try:
-                book.pubdate = datetime.strptime(to_save["pubdate"], "%Y-%m-%d")
-            except ValueError as e:
-                book.pubdate = db.Books.DEFAULT_PUBDATE
-                flash(str(e), category="error")
-                edit_error = True
-        else:
-            book.pubdate = db.Books.DEFAULT_PUBDATE
-
-        if modify_date:
-            book.last_modified = datetime.utcnow()
-            kobo_sync_status.remove_synced_book(edited_books_id, all=True)
-            calibre_db.set_metadata_dirty(book.id)
-
-        calibre_db.session.merge(book)
-        calibre_db.session.commit()
-        if config.config_use_google_drive:
-            gdriveutils.updateGdriveCalibreFromLocal()
-        if meta is not False \
-            and edit_error is not True \
-                and title_author_error is not True \
-                and cover_upload_success is not False:
-            flash(_("Metadata successfully updated"), category="success")
-        if "detail_view" in to_save:
-            return redirect(url_for('web.show_book', book_id=book.id))
-        else:
-            return render_edit_book(book_id)
-    except ValueError as e:
-        log.error_or_exception("Error: {}".format(e))
-        calibre_db.session.rollback()
-        flash(str(e), category="error")
-        return redirect(url_for('web.show_book', book_id=book.id))
-    except (OperationalError, IntegrityError, StaleDataError, InterfaceError) as e:
-        log.error_or_exception("Database error: {}".format(e))
-        calibre_db.session.rollback()
-        flash(_("Oops! Database Error: %(error)s.", error=e.orig if hasattr(e, "orig") else e), category="error")
-        return redirect(url_for('web.show_book', book_id=book.id))
-    except Exception as ex:
-        log.error_or_exception(ex)
-        calibre_db.session.rollback()
-        flash(_("Error editing book: {}".format(ex)), category="error")
-        return redirect(url_for('web.show_book', book_id=book.id))
+    return do_edit_book(book_id)
 
 
 @editbook.route("/upload", methods=["POST"])
 @login_required_if_no_ano
 @upload_required
 def upload():
-    if not config.config_uploading:
-        abort(404)
-    if request.method == 'POST' and 'btn-upload' in request.files:
+    if len(request.files.getlist("btn-upload-format")):
+        book_id = request.form.get('book_id', -1)
+        return do_edit_book(book_id, request.files.getlist("btn-upload-format"))
+    elif len(request.files.getlist("btn-upload")):
         for requested_file in request.files.getlist("btn-upload"):
             try:
                 modify_date = False
                 # create the function for sorting...
-                calibre_db.update_title_sort(config)
-                calibre_db.session.connection().connection.connection.create_function('uuid4', 0, lambda: str(uuid4()))
-
+                calibre_db.create_functions(config)
                 meta, error = file_handling_on_upload(requested_file)
                 if error:
                     return error
@@ -275,9 +140,7 @@ def upload():
                                                         input_authors[0],
                                                         meta.file_path,
                                                         title_dir + meta.extension.lower())
-
                 move_coverfile(meta, db_book)
-
                 if modify_date:
                     calibre_db.set_metadata_dirty(book_id)
                 # save data to database, reread data
@@ -305,6 +168,7 @@ def upload():
                 flash(_("Oops! Database Error: %(error)s.", error=e.orig if hasattr(e, "orig") else e),
                       category="error")
         return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')
+    abort(404)
 
 
 @editbook.route("/admin/book/convert/<int:book_id>", methods=['POST'])
@@ -440,7 +304,7 @@ def edit_list_book(param):
                                mimetype='application/json')
         else:
             return _("Parameter not found"), 400
-        book.last_modified = datetime.utcnow()
+        book.last_modified = datetime.now(timezone.utc)
 
         calibre_db.session.commit()
         # revert change for sort if automatic fields link is deactivated
@@ -556,7 +420,7 @@ def table_xchange_author_title():
                 # toDo: Handle error
                 edit_error = helper.update_dir_structure(edited_books_id, config.get_book_path(), input_authors[0])
             if modify_date:
-                book.last_modified = datetime.utcnow()
+                book.last_modified = datetime.now(timezone.utc)
                 calibre_db.set_metadata_dirty(book.id)
             try:
                 calibre_db.session.commit()
@@ -571,23 +435,167 @@ def table_xchange_author_title():
     return ""
 
 
-def merge_metadata(to_save, meta):
-    if to_save.get('author_name', "") == _('Unknown'):
-        to_save['author_name'] = ''
-    if to_save.get('book_title', "") == _('Unknown'):
-        to_save['book_title'] = ''
-    if not to_save["languages"] and meta.languages:
-        upload_language = True
-    else:
-        upload_language = False
+def do_edit_book(book_id, upload_formats=None):
+    modify_date = False
+    edit_error = False
+
+    # create the function for sorting...
+    calibre_db.create_functions(config)
+
+    book = calibre_db.get_filtered_book(book_id, allow_show_archived=True)
+    # Book not found
+    if not book:
+        flash(_("Oops! Selected book is unavailable. File does not exist or is not accessible"),
+              category="error")
+        return redirect(url_for("web.index"))
+
+    to_save = request.form.to_dict()
+
+    try:
+        # Update folder of book on local disk
+        edited_books_id = None
+        title_author_error = None
+        # upload_mode = False
+        # handle book title change
+        if "title" in to_save:
+            title_change = handle_title_on_edit(book, to_save["title"])
+        # handle book author change
+        if not upload_formats:
+            input_authors, author_change = handle_author_on_edit(book, to_save["authors"])
+            if author_change or title_change:
+                edited_books_id = book.id
+                modify_date = True
+                title_author_error = helper.update_dir_structure(edited_books_id,
+                                                                 config.get_book_path(),
+                                                                 input_authors[0])
+            if title_author_error:
+                flash(title_author_error, category="error")
+                calibre_db.session.rollback()
+                book = calibre_db.get_filtered_book(book_id, allow_show_archived=True)
+
+            # handle book ratings
+            modify_date |= edit_book_ratings(to_save, book)
+        else:
+            # handle upload other formats from local disk
+            to_save, edit_error = upload_book_formats(upload_formats, book, book_id, book.has_cover)
+        # handle upload covers from local disk
+        cover_upload_success = upload_cover(request, book)
+        if cover_upload_success or to_save.get("format_cover"):
+            book.has_cover = 1
+            modify_date = True
+
+        # upload new covers or new file formats to google drive
+        if config.config_use_google_drive:
+            gdriveutils.updateGdriveCalibreFromLocal()
+
+        if to_save.get("cover_url",):
+            if not current_user.role_upload():
+                edit_error = True
+                flash(_("User has no rights to upload cover"), category="error")
+            if to_save["cover_url"].endswith('/static/generic_cover.jpg'):
+                book.has_cover = 0
+            else:
+                result, error = helper.save_cover_from_url(to_save["cover_url"].strip(), book.path)
+                if result is True:
+                    book.has_cover = 1
+                    modify_date = True
+                    helper.replace_cover_thumbnail_cache(book.id)
+                else:
+                    edit_error = True
+                    flash(error, category="error")
+
+        # Add default series_index to book
+        modify_date |= edit_book_series_index(to_save.get("series_index"), book)
+        # Handle book comments/description
+        modify_date |= edit_book_comments(Markup(to_save.get('comments')).unescape(), book)
+        # Handle identifiers
+        input_identifiers = identifier_list(to_save, book)
+        modification, warning = modify_identifiers(input_identifiers, book.identifiers, calibre_db.session)
+        if warning:
+            flash(_("Identifiers are not Case Sensitive, Overwriting Old Identifier"), category="warning")
+        modify_date |= modification
+        # Handle book tags
+        modify_date |= edit_book_tags(to_save.get('tags'), book)
+        # Handle book series
+        modify_date |= edit_book_series(to_save.get("series"), book)
+        # handle book publisher
+        modify_date |= edit_book_publisher(to_save.get('publisher'), book)
+        # handle book languages
+        try:
+            invalid = []
+            modify_date |= edit_book_languages(to_save.get('languages'), book, upload_mode=upload_formats,
+                                               invalid=invalid)
+            if invalid:
+                for lang in invalid:
+                    flash(_("'%(langname)s' is not a valid language", langname=lang), category="warning")
+        except ValueError as e:
+            flash(str(e), category="error")
+            edit_error = True
+        # handle cc data
+        modify_date |= edit_all_cc_data(book_id, book, to_save)
+
+        if to_save.get("pubdate") is not None:
+            if to_save.get("pubdate"):
+                try:
+                    book.pubdate = datetime.strptime(to_save["pubdate"], "%Y-%m-%d")
+                except ValueError as e:
+                    book.pubdate = db.Books.DEFAULT_PUBDATE
+                    flash(str(e), category="error")
+                    edit_error = True
+            else:
+                book.pubdate = db.Books.DEFAULT_PUBDATE
+
+        if modify_date:
+            book.last_modified = datetime.now(timezone.utc)
+            kobo_sync_status.remove_synced_book(edited_books_id, all=True)
+            calibre_db.set_metadata_dirty(book.id)
+
+        calibre_db.session.merge(book)
+        calibre_db.session.commit()
+        if config.config_use_google_drive:
+            gdriveutils.updateGdriveCalibreFromLocal()
+        if edit_error is not True and title_author_error is not True and cover_upload_success is not False:
+            flash(_("Metadata successfully updated"), category="success")
+
+        if upload_formats:
+            resp = {"location": url_for('edit-book.show_edit_book', book_id=book_id)}
+            return Response(json.dumps(resp), mimetype='application/json')
+
+        if "detail_view" in to_save:
+            return redirect(url_for('web.show_book', book_id=book.id))
+        else:
+            return render_edit_book(book_id)
+    except ValueError as e:
+        log.error_or_exception("Error: {}".format(e))
+        calibre_db.session.rollback()
+        flash(str(e), category="error")
+        return redirect(url_for('web.show_book', book_id=book.id))
+    except (OperationalError, IntegrityError, StaleDataError, InterfaceError) as e:
+        log.error_or_exception("Database error: {}".format(e))
+        calibre_db.session.rollback()
+        flash(_("Oops! Database Error: %(error)s.", error=e.orig if hasattr(e, "orig") else e), category="error")
+        return redirect(url_for('web.show_book', book_id=book.id))
+    except Exception as ex:
+        log.error_or_exception(ex)
+        calibre_db.session.rollback()
+        flash(_("Error editing book: {}".format(ex)), category="error")
+        return redirect(url_for('web.show_book', book_id=book.id))
+
+
+def merge_metadata(book, meta, to_save):
+    if meta.cover:
+        to_save['cover_format'] = meta.cover
     for s_field, m_field in [
-            ('tags', 'tags'), ('author_name', 'author'), ('series', 'series'),
+            ('tags', 'tags'), ('authors', 'author'), ('series', 'series'),
             ('series_index', 'series_id'), ('languages', 'languages'),
-            ('book_title', 'title')]:
-        to_save[s_field] = to_save[s_field] or getattr(meta, m_field, '')
-    to_save["description"] = to_save["description"] or Markup(
-        getattr(meta, 'description', '')).unescape()
-    return upload_language
+            ('title', 'title'), ('comments', 'description')]:
+        try:
+            val = None if len(getattr(book, s_field)) else getattr(meta, m_field, '')
+        except TypeError:
+            val = None if len(str(getattr(book, s_field))) else getattr(meta, m_field, '')
+        if val:
+            to_save[s_field] = val
+
 
 def identifier_list(to_save, book):
     """Generate a list of Identifiers from form information"""
@@ -707,8 +715,8 @@ def create_book_on_upload(modify_date, meta):
         pubdate = datetime(101, 1, 1)
 
     # Calibre adds books with utc as timezone
-    db_book = db.Books(title, "", sort_authors, datetime.utcnow(), pubdate,
-                       '1', datetime.utcnow(), path, meta.cover, db_author, [], "")
+    db_book = db.Books(title, "", sort_authors, datetime.now(timezone.utc), pubdate,
+                       '1', datetime.now(timezone.utc), path, meta.cover, db_author, [], "")
 
     modify_date |= modify_database_object(input_authors, db_book.authors, db.Authors, calibre_db.session,
                                           'author')
@@ -975,7 +983,7 @@ def render_edit_book(book_id):
 
 def edit_book_ratings(to_save, book):
     changed = False
-    if to_save.get("rating", "").strip():
+    if strip_whitespaces(to_save.get("rating", "")):
         old_rating = False
         if len(book.ratings) > 0:
             old_rating = book.ratings[0].rating
@@ -998,84 +1006,93 @@ def edit_book_ratings(to_save, book):
 
 
 def edit_book_tags(tags, book):
-    input_tags = tags.split(',')
-    input_tags = list(map(lambda it: it.strip(), input_tags))
-    # Remove duplicates
-    input_tags = helper.uniq(input_tags)
-    return modify_database_object(input_tags, book.tags, db.Tags, calibre_db.session, 'tags')
-
+    if tags is not None:
+        input_tags = tags.split(',')
+        input_tags = list(map(lambda it: strip_whitespaces(it), input_tags))
+        # Remove duplicates
+        input_tags = helper.uniq(input_tags)
+        return modify_database_object(input_tags, book.tags, db.Tags, calibre_db.session, 'tags')
+    return False
 
 def edit_book_series(series, book):
-    input_series = [series.strip()]
-    input_series = [x for x in input_series if x != '']
-    return modify_database_object(input_series, book.series, db.Series, calibre_db.session, 'series')
+    if series is not None:
+        input_series = [strip_whitespaces(series)]
+        input_series = [x for x in input_series if x != '']
+        return modify_database_object(input_series, book.series, db.Series, calibre_db.session, 'series')
+    return False
 
 
 def edit_book_series_index(series_index, book):
-    # Add default series_index to book
-    modify_date = False
-    series_index = series_index or '1'
-    if not series_index.replace('.', '', 1).isdigit():
-        flash(_("%(seriesindex)s is not a valid number, skipping", seriesindex=series_index), category="warning")
-        return False
-    if str(book.series_index) != series_index:
-        book.series_index = series_index
-        modify_date = True
-    return modify_date
+    if series_index:
+        # Add default series_index to book
+        modify_date = False
+        series_index = series_index or '1'
+        if not series_index.replace('.', '', 1).isdigit():
+            flash(_("Seriesindex: %(seriesindex)s is not a valid number, skipping", seriesindex=series_index), category="warning")
+            return False
+        if str(book.series_index) != series_index:
+            book.series_index = series_index
+            modify_date = True
+        return modify_date
+    return False
 
 
 # Handle book comments/description
 def edit_book_comments(comments, book):
-    modify_date = False
-    if comments:
-        comments = clean_string(comments, book.id)
-    if len(book.comments):
-        if book.comments[0].text != comments:
-            book.comments[0].text = comments
-            modify_date = True
-    else:
+    if comments is not None:
+        modify_date = False
         if comments:
-            book.comments.append(db.Comments(comment=comments, book=book.id))
-            modify_date = True
-    return modify_date
+            comments = clean_string(comments, book.id)
+        if len(book.comments):
+            if book.comments[0].text != comments:
+                book.comments[0].text = comments
+                modify_date = True
+        else:
+            if comments:
+                book.comments.append(db.Comments(comment=comments, book=book.id))
+                modify_date = True
+        return modify_date
 
 
 def edit_book_languages(languages, book, upload_mode=False, invalid=None):
-    input_languages = languages.split(',')
-    unknown_languages = []
-    if not upload_mode:
-        input_l = isoLanguages.get_language_codes(get_locale(), input_languages, unknown_languages)
-    else:
-        input_l = isoLanguages.get_valid_language_codes(get_locale(), input_languages, unknown_languages)
-    for lang in unknown_languages:
-        log.error("'%s' is not a valid language", lang)
-        if isinstance(invalid, list):
-            invalid.append(lang)
+    if languages is not None:
+        input_languages = languages.split(',')
+        unknown_languages = []
+        if not upload_mode:
+            input_l = isoLanguages.get_language_code_from_name(get_locale(), input_languages, unknown_languages)
         else:
-            raise ValueError(_("'%(langname)s' is not a valid language", langname=lang))
-    # ToDo: Not working correct
-    if upload_mode and len(input_l) == 1:
-        # If the language of the file is excluded from the users view, it's not imported, to allow the user to view
-        # the book it's language is set to the filter language
-        if input_l[0] != current_user.filter_language() and current_user.filter_language() != "all":
-            input_l[0] = calibre_db.session.query(db.Languages). \
-                filter(db.Languages.lang_code == current_user.filter_language()).first().lang_code
-    # Remove duplicates
-    input_l = helper.uniq(input_l)
-    return modify_database_object(input_l, book.languages, db.Languages, calibre_db.session, 'languages')
+            input_l = isoLanguages.get_valid_language_codes_from_code(get_locale(), input_languages, unknown_languages)
+        for lang in unknown_languages:
+            log.error("'%s' is not a valid language", lang)
+            if isinstance(invalid, list):
+                invalid.append(lang)
+            else:
+                raise ValueError(_("'%(langname)s' is not a valid language", langname=lang))
+        # ToDo: Not working correct
+        if upload_mode and len(input_l) == 1:
+            # If the language of the file is excluded from the users view, it's not imported, to allow the user to view
+            # the book it's language is set to the filter language
+            if input_l[0] != current_user.filter_language() and current_user.filter_language() != "all":
+                input_l[0] = calibre_db.session.query(db.Languages). \
+                    filter(db.Languages.lang_code == current_user.filter_language()).first().lang_code
+        # Remove duplicates from normalized langcodes
+        input_l = helper.uniq(input_l)
+        return modify_database_object(input_l, book.languages, db.Languages, calibre_db.session, 'languages')
+    return False
 
 
 def edit_book_publisher(publishers, book):
-    changed = False
-    if publishers:
-        publisher = publishers.rstrip().strip()
-        if len(book.publishers) == 0 or (len(book.publishers) > 0 and publisher != book.publishers[0].name):
-            changed |= modify_database_object([publisher], book.publishers, db.Publishers, calibre_db.session,
-                                              'publisher')
-    elif len(book.publishers):
-        changed |= modify_database_object([], book.publishers, db.Publishers, calibre_db.session, 'publisher')
-    return changed
-
+    if publishers is not None:
+        changed = False
+        if publishers:
+            publisher = strip_whitespaces(publishers)
+            if len(book.publishers) == 0 or (len(book.publishers) > 0 and publisher != book.publishers[0].name):
+                changed |= modify_database_object([publisher], book.publishers, db.Publishers, calibre_db.session,
+                                                  'publisher')
+        elif len(book.publishers):
+            changed |= modify_database_object([], book.publishers, db.Publishers, calibre_db.session, 'publisher')
+        return changed
+    return False
 
 def edit_cc_data_value(book_id, book, c, to_save, cc_db_value, cc_string):
     changed = False
@@ -1115,7 +1132,7 @@ def edit_cc_data_string(book, c, to_save, cc_db_value, cc_string):
     changed = False
     if c.datatype == 'rating':
         to_save[cc_string] = str(int(float(to_save[cc_string]) * 2))
-    if to_save[cc_string].strip() != cc_db_value:
+    if strip_whitespaces(to_save[cc_string]) != cc_db_value:
         if cc_db_value is not None:
             # remove old cc_val
             del_cc = getattr(book, cc_string)[0]
@@ -1125,15 +1142,15 @@ def edit_cc_data_string(book, c, to_save, cc_db_value, cc_string):
                 changed = True
         cc_class = db.cc_classes[c.id]
         new_cc = calibre_db.session.query(cc_class).filter(
-            cc_class.value == to_save[cc_string].strip()).first()
+            cc_class.value == strip_whitespaces(to_save[cc_string])).first()
         # if no cc val is found add it
         if new_cc is None:
-            new_cc = cc_class(value=to_save[cc_string].strip())
+            new_cc = cc_class(value=strip_whitespaces(to_save[cc_string]))
             calibre_db.session.add(new_cc)
             changed = True
             calibre_db.session.flush()
             new_cc = calibre_db.session.query(cc_class).filter(
-                cc_class.value == to_save[cc_string].strip()).first()
+                cc_class.value == strip_whitespaces(to_save[cc_string])).first()
         # add cc value to book
         getattr(book, cc_string).append(new_cc)
     return changed, to_save
@@ -1156,61 +1173,66 @@ def edit_cc_data(book_id, book, to_save, cc):
     changed = False
     for c in cc:
         cc_string = "custom_column_" + str(c.id)
-        if not c.is_multiple:
-            if len(getattr(book, cc_string)) > 0:
-                cc_db_value = getattr(book, cc_string)[0].value
-            else:
-                cc_db_value = None
-            if to_save[cc_string].strip():
-                if c.datatype in ['int', 'bool', 'float', "datetime", "comments"]:
-                    change, to_save = edit_cc_data_value(book_id, book, c, to_save, cc_db_value, cc_string)
+        if to_save.get(cc_string) is not None:
+            if not c.is_multiple:
+                if len(getattr(book, cc_string)) > 0:
+                    cc_db_value = getattr(book, cc_string)[0].value
                 else:
-                    change, to_save = edit_cc_data_string(book, c, to_save, cc_db_value, cc_string)
-                changed |= change
+                    cc_db_value = None
+                if strip_whitespaces(to_save[cc_string]):
+                    if c.datatype in ['int', 'bool', 'float', "datetime", "comments"]:
+                        change, to_save = edit_cc_data_value(book_id, book, c, to_save, cc_db_value, cc_string)
+                    else:
+                        change, to_save = edit_cc_data_string(book, c, to_save, cc_db_value, cc_string)
+                    changed |= change
+                else:
+                    if cc_db_value is not None:
+                        # remove old cc_val
+                        del_cc = getattr(book, cc_string)[0]
+                        getattr(book, cc_string).remove(del_cc)
+                        if not del_cc.books or len(del_cc.books) == 0:
+                            calibre_db.session.delete(del_cc)
+                            changed = True
             else:
-                if cc_db_value is not None:
-                    # remove old cc_val
-                    del_cc = getattr(book, cc_string)[0]
-                    getattr(book, cc_string).remove(del_cc)
-                    if not del_cc.books or len(del_cc.books) == 0:
-                        calibre_db.session.delete(del_cc)
-                        changed = True
-        else:
-            input_tags = to_save[cc_string].split(',')
-            input_tags = list(map(lambda it: it.strip(), input_tags))
-            changed |= modify_database_object(input_tags,
-                                              getattr(book, cc_string),
-                                              db.cc_classes[c.id],
-                                              calibre_db.session,
-                                              'custom')
+                input_tags = to_save[cc_string].split(',')
+                input_tags = list(map(lambda it: strip_whitespaces(it), input_tags))
+                changed |= modify_database_object(input_tags,
+                                                  getattr(book, cc_string),
+                                                  db.cc_classes[c.id],
+                                                  calibre_db.session,
+                                                  'custom')
     return changed
 
 
-# returns None if no file is uploaded
-# returns False if an error occurs, in all other cases the ebook metadata is returned
-def upload_single_file(file_request, book, book_id):
+# returns False if an error occurs or no book is uploaded, in all other cases the ebook metadata to change is returned
+def upload_book_formats(requested_files, book, book_id, no_cover=True):
     # Check and handle Uploaded file
-    requested_file = file_request.files.get('btn-upload-format', None)
+    to_save = dict()
+    error = False
     allowed_extensions = config.config_upload_formats.split(',')
-    if requested_file:
+    for requested_file in requested_files:
+        current_filename = requested_file.filename
         if config.config_check_extensions and allowed_extensions != ['']:
             if not validate_mime_type(requested_file, allowed_extensions):
                 flash(_("File type isn't allowed to be uploaded to this server"), category="error")
-                return False
-        # check for empty request
-        if requested_file.filename != '':
+                error = True
+                continue
+        if current_filename != '':
             if not current_user.role_upload():
                 flash(_("User has no rights to upload additional file formats"), category="error")
-                return False
-            if '.' in requested_file.filename:
-                file_ext = requested_file.filename.rsplit('.', 1)[-1].lower()
+                error = True
+                continue
+            if '.' in current_filename:
+                file_ext = current_filename.rsplit('.', 1)[-1].lower()
                 if file_ext not in allowed_extensions and '' not in allowed_extensions:
                     flash(_("File extension '%(ext)s' is not allowed to be uploaded to this server", ext=file_ext),
                           category="error")
-                    return False
+                    error = True
+                    continue
             else:
                 flash(_('File to be uploaded must have an extension'), category="error")
-                return False
+                error = True
+                continue
 
             file_name = book.path.rsplit('/', 1)[-1]
             filepath = os.path.normpath(os.path.join(config.get_book_path(), book.path))
@@ -1223,41 +1245,50 @@ def upload_single_file(file_request, book, book_id):
                 except OSError:
                     flash(_("Failed to create path %(path)s (Permission denied).", path=filepath),
                           category="error")
-                    return False
+                    error = True
+                    continue
             try:
                 requested_file.save(saved_filename)
             except OSError:
                 flash(_("Failed to store file %(file)s.", file=saved_filename), category="error")
-                return False
+                error = True
+                continue
 
             file_size = os.path.getsize(saved_filename)
-            is_format = calibre_db.get_book_format(book_id, file_ext.upper())
 
             # Format entry already exists, no need to update the database
-            if is_format:
+            if calibre_db.get_book_format(book_id, file_ext.upper()):
                 log.warning('Book format %s already existing', file_ext.upper())
             else:
                 try:
                     db_format = db.Data(book_id, file_ext.upper(), file_size, file_name)
                     calibre_db.session.add(db_format)
                     calibre_db.session.commit()
-                    calibre_db.update_title_sort(config)
+                    calibre_db.create_functions(config)
                 except (OperationalError, IntegrityError, StaleDataError) as e:
                     calibre_db.session.rollback()
                     log.error_or_exception("Database error: {}".format(e))
                     flash(_("Oops! Database Error: %(error)s.", error=e.orig if hasattr(e, "orig") else e),
                           category="error")
-                    return False  # return redirect(url_for('web.show_book', book_id=book.id))
+                    error = True
+                    continue
 
             # Queue uploader info
             link = '<a href="{}">{}</a>'.format(url_for('web.show_book', book_id=book.id), escape(book.title))
             upload_text = N_("File format %(ext)s added to %(book)s", ext=file_ext.upper(), book=link)
             WorkerThread.add(current_user.name, TaskUpload(upload_text, escape(book.title)))
-
-            return uploader.process(
-                saved_filename, *os.path.splitext(requested_file.filename),
-                rar_executable=config.config_rarfile_location)
-    return None
+            meta = uploader.process(
+                saved_filename,
+                *os.path.splitext(current_filename),
+                rar_executable=config.config_rarfile_location,
+                no_cover=no_cover)
+            merge_metadata(book, meta, to_save)
+    #if to_save.get('languages'):
+    #    langs = []
+    #    for lang_code in to_save['languages'].split(','):
+    #        langs.append(isoLanguages.get_language_name(get_locale(), lang_code))
+    #    to_save['languages'] = ",".join(langs)
+    return to_save, error
 
 
 def upload_cover(cover_request, book):
@@ -1280,7 +1311,7 @@ def upload_cover(cover_request, book):
 
 def handle_title_on_edit(book, book_title):
     # handle book title
-    book_title = book_title.rstrip().strip()
+    book_title = strip_whitespaces(book_title)
     if book.title != book_title:
         if book_title == '':
             book_title = _(u'Unknown')
@@ -1291,7 +1322,6 @@ def handle_title_on_edit(book, book_title):
 
 def handle_author_on_edit(book, author_name, update_stored=True):
     change = False
-    # handle author(s)
     input_authors = prepare_authors(author_name, config.get_book_path(), config.config_use_google_drive)
 
     # Search for each author if author is in database, if not, author name and sorted author name is generated new
@@ -1321,7 +1351,6 @@ def search_objects_remove(db_book_object, db_type, input_elements):
         if db_type == 'custom':
             type_elements = c_elements.value
         else:
-            # type_elements = c_elements.name
             type_elements = c_elements
         for inp_element in input_elements:
             if type_elements == inp_element:

cps/embed_helper.py

@@ -35,9 +35,7 @@ def do_calibre_export(book_id, book_format):
         my_env = os.environ.copy()
         if config.config_calibre_split:
             my_env['CALIBRE_OVERRIDE_DATABASE_PATH'] = os.path.join(config.config_calibre_dir, "metadata.db")
-            library_path = config.config_calibre_split_dir
-        else:
-            library_path = config.config_calibre_dir
+        library_path = config.get_book_path()
         opf_command = [calibredb_binarypath, 'export', '--dont-write-opf', '--with-library', library_path,
                        '--to-dir', tmp_dir, '--formats', book_format, "--template", "{}".format(temp_file_name),
                        str(book_id)]

cps/epub.py

@@ -25,6 +25,7 @@ from . import config, logger
 from .helper import split_authors
 from .epub_helper import get_content_opf, default_ns
 from .constants import BookMeta
+from .string_helper import strip_whitespaces
 
 log = logger.create()
 
@@ -65,7 +66,7 @@ def get_epub_layout(book, book_data):
         return layout[0]
 
 
-def get_epub_info(tmp_file_path, original_file_name, original_file_extension):
+def get_epub_info(tmp_file_path, original_file_name, original_file_extension, no_cover_processing):
     ns = {
         'n': 'urn:oasis:names:tc:opendocument:xmlns:container',
         'pkg': 'http://www.idpf.org/2007/opf',
@@ -90,7 +91,7 @@ def get_epub_info(tmp_file_path, original_file_name, original_file_extension):
             elif s == 'date':
                 epub_metadata[s] = tmp[0][:10]
             else:
-                epub_metadata[s] = tmp[0].strip()
+                epub_metadata[s] = strip_whitespaces(tmp[0])
         else:
             epub_metadata[s] = 'Unknown'
 
@@ -116,7 +117,10 @@ def get_epub_info(tmp_file_path, original_file_name, original_file_extension):
     epub_metadata = parse_epub_series(ns, tree, epub_metadata)
 
     epub_zip = zipfile.ZipFile(tmp_file_path)
-    cover_file = parse_epub_cover(ns, tree, epub_zip, cover_path, tmp_file_path)
+    if not no_cover_processing:
+        cover_file = parse_epub_cover(ns, tree, epub_zip, cover_path, tmp_file_path)
+    else:
+        cover_file = None
 
     identifiers = []
     for node in p.xpath('dc:identifier', namespaces=ns):

cps/helper.py

@@ -25,7 +25,7 @@ import re
 import regex
 import shutil
 import socket
-from datetime import datetime, timedelta
+from datetime import datetime, timedelta, timezone
 import requests
 import unidecode
 from uuid import uuid4
@@ -43,15 +43,16 @@ from markupsafe import escape
 from urllib.parse import quote
 
 try:
-    import advocate
-    from advocate.exceptions import UnacceptableAddressException
+    from . import cw_advocate
+    from .cw_advocate.exceptions import UnacceptableAddressException
     use_advocate = True
-except ImportError:
+except ImportError as e:
     use_advocate = False
     advocate = requests
     UnacceptableAddressException = MissingSchema = BaseException
 
 from . import calibre_db, cli_param
+from .string_helper import strip_whitespaces
 from .tasks.convert import TaskConvert
 from . import logger, config, db, ub, fs
 from . import gdriveutils as gd
@@ -118,7 +119,7 @@ def convert_book_format(book_id, calibre_path, old_book_format, new_book_format,
 # Texts are not lazy translated as they are supposed to get send out as is
 def send_test_mail(ereader_mail, user_name):
     for email in ereader_mail.split(','):
-        email = email.strip()
+        email = strip_whitespaces(email)
         WorkerThread.add(user_name, TaskEmail(_('Calibre-Web Test Email'), None, None,
                          config.get_mail_settings(), email, N_("Test Email"),
                                               _('This Email has been sent via Calibre-Web.')))
@@ -228,7 +229,7 @@ def send_mail(book_id, book_format, convert, ereader_mail, calibrepath, user_id)
             link = '<a href="{}">{}</a>'.format(url_for('web.show_book', book_id=book_id), escape(book.title))
             email_text = N_("%(book)s send to eReader", book=link)
             for email in ereader_mail.split(','):
-                email = email.strip()
+                email = strip_whitespaces(email)
                 WorkerThread.add(user_id, TaskEmail(_("Send to eReader"), book.path, converted_file_name,
                                  config.get_mail_settings(), email,
                                  email_text, _('This Email has been sent via Calibre-Web.'), book.id))
@@ -252,7 +253,7 @@ def get_valid_filename(value, replace_whitespace=True, chars=128):
         # pipe has to be replaced with comma
         value = re.sub(r'[|]+', ',', value, flags=re.U)
 
-    value = value.encode('utf-8')[:chars].decode('utf-8', errors='ignore').strip()
+    value = strip_whitespaces(value.encode('utf-8')[:chars].decode('utf-8', errors='ignore'))
 
     if not value:
         raise ValueError("Filename cannot be empty")
@@ -267,11 +268,11 @@ def split_authors(values):
             commas = author.count(',')
             if commas == 1:
                 author_split = author.split(',')
-                authors_list.append(author_split[1].strip() + ' ' + author_split[0].strip())
+                authors_list.append(strip_whitespaces(author_split[1]) + ' ' + strip_whitespaces(author_split[0]))
             elif commas > 1:
-                authors_list.extend([x.strip() for x in author.split(',')])
+                authors_list.extend([strip_whitespaces(x) for x in author.split(',')])
             else:
-                authors_list.append(author.strip())
+                authors_list.append(strip_whitespaces(author))
     return authors_list
 
 
@@ -327,7 +328,7 @@ def edit_book_read_status(book_id, read_status=None):
         ub.session_commit("Book {} readbit toggled".format(book_id))
     else:
         try:
-            calibre_db.update_title_sort(config)
+            calibre_db.create_functions(config)
             book = calibre_db.get_filtered_book(book_id)
             book_read_status = getattr(book, 'custom_column_' + str(config.config_read_column))
             if len(book_read_status):
@@ -417,8 +418,6 @@ def rename_author_path(first_author, old_author_dir, renamed_author, calibre_pat
     # Create new_author_dir from parameter or from database
     # Create new title_dir from database and add id
     new_authordir = get_valid_filename(first_author, chars=96)
-    # new_author = calibre_db.session.query(db.Authors).filter(db.Authors.name == renamed_author).first()
-    # old_author_dir = get_valid_filename(old_author_name, chars=96)
     new_author_rename_dir = get_valid_filename(renamed_author, chars=96)
     if gdrive:
         g_file = gd.getFileFromEbooksFolder(None, old_author_dir)
@@ -467,7 +466,6 @@ def update_dir_structure_file(book_id, calibre_path, original_filepath, new_auth
                                      db_filename,
                                      original_filepath,
                                      path)
-        # old_path = os.path.join(calibre_path, author_dir, new_title_dir).replace('\\', '/')
         new_path = os.path.join(calibre_path, new_author_dir, new_title_dir).replace('\\', '/')
         all_new_name = get_valid_filename(local_book.title, chars=42) + ' - ' \
                        + get_valid_filename(new_author, chars=42)
@@ -476,8 +474,6 @@ def update_dir_structure_file(book_id, calibre_path, original_filepath, new_auth
 
         if error:
             return error
-
-    # Rename all files from old names to new names
     return False
 
 
@@ -489,7 +485,7 @@ def upload_new_file_gdrive(book_id, first_author, title, title_dir, original_fil
                                title_dir + " (" + str(book_id) + ")")
     book.path = gdrive_path.replace("\\", "/")
     gd.uploadFileToEbooksFolder(os.path.join(gdrive_path, file_name).replace("\\", "/"), original_filepath)
-    return False # rename_files_on_change(first_author, renamed_author, local_book=book, gdrive=True)
+    return False
 
 
 def update_dir_structure_gdrive(book_id, first_author):
@@ -522,19 +518,21 @@ def update_dir_structure_gdrive(book_id, first_author):
         all_new_name = get_valid_filename(book.title, chars=42) + ' - ' \
                        + get_valid_filename(new_authordir, chars=42)
         rename_all_files_on_change(book, book.path, book.path, all_new_name, gdrive=True)  # todo: Move filenames on gdrive
-    # change location in database to new author/title path
-    # book.path = os.path.join(authordir, new_titledir).replace('\\', '/')
     return False
 
 
 def move_files_on_change(calibre_path, new_author_dir, new_titledir, localbook, db_filename, original_filepath, path):
     new_path = os.path.join(calibre_path, new_author_dir, new_titledir)
-    # new_name = get_valid_filename(localbook.title, chars=96) + ' - ' + new_author_dir
     try:
         if original_filepath:
             if not os.path.isdir(new_path):
                 os.makedirs(new_path)
-            shutil.move(original_filepath, os.path.join(new_path, db_filename))
+            try:
+                shutil.move(original_filepath, os.path.join(new_path, db_filename))
+            except OSError:
+                log.error("Rename title from {} to {} failed with error, trying to "
+                          "move without metadata".format(path, new_path))
+                shutil.move(original_filepath, os.path.join(new_path, db_filename), copy_function=shutil.copy)
             log.debug("Moving title: %s to %s", original_filepath, new_path)
         else:
             # Check new path is not valid path
@@ -661,7 +659,7 @@ def check_email(email):
 
 
 def check_username(username):
-    username = username.strip()
+    username = strip_whitespaces(username)
     if ub.session.query(ub.User).filter(func.lower(ub.User.name) == username.lower()).scalar():
         log.error("This username is already taken")
         raise Exception(_("This username is already taken"))
@@ -669,16 +667,18 @@ def check_username(username):
 
 
 def valid_email(emails):
+    valid_emails = []
     for email in emails.split(','):
-	    email = email.strip()
-	    # if email is not deleted
-	    if email:
-	        # Regex according to https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/email#validation
-	        if not re.search(r"^[\w.!#$%&'*+\\/=?^_`{|}~-]+@[\w](?:[\w-]{0,61}[\w])?(?:\.[\w](?:[\w-]{0,61}[\w])?)*$",
-	                         email):
-	            log.error("Invalid Email address format")
-	            raise Exception(_("Invalid Email address format"))
-    return email
+        email = strip_whitespaces(email)
+        # if email is not deleted
+        if email:
+            # Regex according to https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/email#validation
+            if not re.search(r"^[\w.!#$%&'*+\\/=?^_`{|}~-]+@[\w](?:[\w-]{0,61}[\w])?(?:\.[\w](?:[\w-]{0,61}[\w])?)*$",
+                             email):
+                log.error("Invalid Email address format for {}".format(email))
+                raise Exception(_("Invalid Email address format"))
+            valid_emails.append(email)
+    return ",".join(valid_emails)
 
 
 def valid_password(check_password):
@@ -788,24 +788,23 @@ def get_book_cover_internal(book, resolution=None):
 
 def get_book_cover_thumbnail(book, resolution):
     if book and book.has_cover:
-        return ub.session \
-            .query(ub.Thumbnail) \
-            .filter(ub.Thumbnail.type == THUMBNAIL_TYPE_COVER) \
-            .filter(ub.Thumbnail.entity_id == book.id) \
-            .filter(ub.Thumbnail.resolution == resolution) \
-            .filter(or_(ub.Thumbnail.expiration.is_(None), ub.Thumbnail.expiration > datetime.utcnow())) \
-            .first()
+        return (ub.session
+                .query(ub.Thumbnail)
+                .filter(ub.Thumbnail.type == THUMBNAIL_TYPE_COVER)
+                .filter(ub.Thumbnail.entity_id == book.id)
+                .filter(ub.Thumbnail.resolution == resolution)
+                .filter(or_(ub.Thumbnail.expiration.is_(None), ub.Thumbnail.expiration > datetime.now(timezone.utc)))
+                .first())
 
 
 def get_series_thumbnail_on_failure(series_id, resolution):
-    book = calibre_db.session \
-        .query(db.Books) \
-        .join(db.books_series_link) \
-        .join(db.Series) \
-        .filter(db.Series.id == series_id) \
-        .filter(db.Books.has_cover == 1) \
-        .first()
-
+    book = (calibre_db.session
+        .query(db.Books)
+        .join(db.books_series_link)
+        .join(db.Series)
+        .filter(db.Series.id == series_id)
+        .filter(db.Books.has_cover == 1)
+        .first())
     return get_book_cover_internal(book, resolution=resolution)
 
 
@@ -827,13 +826,13 @@ def get_series_cover_internal(series_id, resolution=None):
 
 
 def get_series_thumbnail(series_id, resolution):
-    return ub.session \
-        .query(ub.Thumbnail) \
-        .filter(ub.Thumbnail.type == THUMBNAIL_TYPE_SERIES) \
-        .filter(ub.Thumbnail.entity_id == series_id) \
-        .filter(ub.Thumbnail.resolution == resolution) \
-        .filter(or_(ub.Thumbnail.expiration.is_(None), ub.Thumbnail.expiration > datetime.utcnow())) \
-        .first()
+    return (ub.session
+        .query(ub.Thumbnail)
+        .filter(ub.Thumbnail.type == THUMBNAIL_TYPE_SERIES)
+        .filter(ub.Thumbnail.entity_id == series_id)
+        .filter(ub.Thumbnail.resolution == resolution)
+        .filter(or_(ub.Thumbnail.expiration.is_(None), ub.Thumbnail.expiration > datetime.now(timezone.utc)))
+        .first())
 
 
 # saves book cover from url
@@ -842,7 +841,7 @@ def save_cover_from_url(url, book_path):
         if cli_param.allow_localhost:
             img = requests.get(url, timeout=(10, 200), allow_redirects=False)  # ToDo: Error Handling
         elif use_advocate:
-            img = advocate.get(url, timeout=(10, 200), allow_redirects=False)      # ToDo: Error Handling
+            img = cw_advocate.get(url, timeout=(10, 200), allow_redirects=False)      # ToDo: Error Handling
         else:
             log.error("python module advocate is not installed but is needed")
             return False, _("Python module 'advocate' is not installed but is needed for cover uploads")

cps/isoLanguages.py

@@ -15,24 +15,17 @@
 #
 #   You should have received a copy of the GNU General Public License
 #   along with this program. If not, see <http://www.gnu.org/licenses/>.
+import sys
 
 from .iso_language_names import LANGUAGE_NAMES as _LANGUAGE_NAMES
 from . import logger
+from .string_helper import strip_whitespaces
 
 log = logger.create()
 
 
 try:
-    from iso639 import languages, __version__
-    get = languages.get
-except ImportError:
     from pycountry import languages as pyc_languages
-    try:
-        import pkg_resources
-        __version__ = pkg_resources.get_distribution('pycountry').version + ' (PyCountry)'
-        del pkg_resources
-    except (ImportError, Exception):
-        __version__ = "? (PyCountry)"
 
     def _copy_fields(l):
         l.part1 = getattr(l, 'alpha_2', None)
@@ -46,6 +39,11 @@ except ImportError:
             return _copy_fields(pyc_languages.get(alpha_2=part1))
         if name is not None:
             return _copy_fields(pyc_languages.get(name=name))
+except ImportError as ex:
+    if sys.version_info >= (3, 12):
+        print("Python 3.12 isn't compatible with iso-639. Please install pycountry.")
+    from iso639 import languages
+    get = languages.get
 
 
 def get_language_names(locale):
@@ -69,20 +67,20 @@ def get_language_name(locale, lang_code):
     return name
 
 
-def get_language_codes(locale, language_names, remainder=None):
-    language_names = set(x.strip().lower() for x in language_names if x)
+def get_language_code_from_name(locale, language_names, remainder=None):
+    language_names = set(strip_whitespaces(x).lower() for x in language_names if x)
     lang = list()
-    for k, v in get_language_names(locale).items():
-        v = v.lower()
-        if v in language_names:
-            lang.append(k)
-            language_names.remove(v)
+    for key, val in get_language_names(locale).items():
+        val = val.lower()
+        if val in language_names:
+            lang.append(key)
+            language_names.remove(val)
     if remainder is not None and language_names:
         remainder.extend(language_names)
     return lang
 
 
-def get_valid_language_codes(locale, language_names, remainder=None):
+def get_valid_language_codes_from_code(locale, language_names, remainder=None):
     lang = list()
     if "" in language_names:
         language_names.remove("")
@@ -103,6 +101,6 @@ def get_lang3(lang):
             ret_value = lang
         else:
             ret_value = ""
-    except KeyError:
+    except (KeyError, AttributeError):
         ret_value = lang
     return ret_value

cps/iso_language_names.py

@@ -8138,6 +8138,384 @@ LANGUAGE_NAMES = {
         "zul": "Zulu",
         "zun": "Zuni"
     },
+    "sl": {
+        "abk": "abhazijski",
+        "ace": "achinese",
+        "ach": "Acoli",
+        "ada": "Adangme",
+        "ady": "Adyghe",
+        "aar": "afarski",
+        "afh": "Afrihili",
+        "afr": "afrikanski",
+        "ain": "Ainu (Japan)",
+        "aka": "Akan",
+        "akk": "akadski",
+        "sqi": "albanščina",
+        "ale": "aleutski",
+        "amh": "amharski",
+        "anp": "Angika",
+        "ara": "arabski",
+        "arg": "aragonski",
+        "arp": "Arapaho",
+        "arw": "araukanski",
+        "hye": "armenščina",
+        "asm": "asamski",
+        "ast": "Asturian",
+        "ava": "avarski",
+        "ave": "avestijski jeziki",
+        "awa": "Awadhi",
+        "aym": "Aymara",
+        "aze": "azerbajdžanski",
+        "ban": "balijščina",
+        "bal": "belučijski",
+        "bam": "bambarski",
+        "bas": "Basa (Cameroon)",
+        "bak": "baškirski",
+        "eus": "baskovščina",
+        "bej": "Beja",
+        "bel": "beloruščina",
+        "bem": "Bemba (Zambia)",
+        "ben": "bengalščina",
+        "bit": "Berinomo",
+        "bho": "Bhojpuri",
+        "bik": "bikolščina",
+        "byn": "Bilin",
+        "bin": "Bini",
+        "bis": "bislama",
+        "zbl": "Blissymbols",
+        "bos": "bošnjaščina",
+        "bra": "Braj",
+        "bre": "bretonščina",
+        "bug": "buginščina",
+        "bul": "bolgarščina",
+        "bua": "burjatščina",
+        "mya": "burmanščina",
+        "cad": "kadajščina?",
+        "cat": "katalonščina",
+        "ceb": "cebuanščina",
+        "chg": "Chagatai",
+        "cha": "čamorščina",
+        "che": "čečenščina",
+        "chr": "čerokeščina",
+        "chy": "čejenščina",
+        "chb": "čibčevščina",
+        "zho": "kitajščina",
+        "chn": "Chinook jargon",
+        "chp": "čipevščina",
+        "cho": "Choctaw",
+        "cht": "Cholón",
+        "chk": "Chuukese",
+        "chv": "čuvaščina",
+        "cop": "koptščina",
+        "cor": "kornijščina",
+        "cos": "korzijščina",
+        "cre": "krijščina",
+        "mus": "Creek",
+        "hrv": "hrvaščina",
+        "ces": "češčina",
+        "dak": "Dakota",
+        "dan": "danski",
+        "dar": "Dargwa",
+        "del": "Delaware",
+        "div": "Dhivehi",
+        "din": "Dinka",
+        "doi": "Dogri (macrolanguage)",
+        "dgr": "Dogrib",
+        "dua": "Duala",
+        "nld": "nizozemščina",
+        "dse": "Dutch Sign Language",
+        "dyu": "Dyula",
+        "dzo": "dzongkha",
+        "efi": "Efik",
+        "egy": "egipčanski",
+        "eka": "Ekajuk",
+        "elx": "elamščina",
+        "eng": "angleščina",
+        "enu": "Enu",
+        "myv": "Erzya",
+        "epo": "esperanto",
+        "est": "estonščina",
+        "ewe": "evenščina",
+        "ewo": "Ewondo",
+        "fan": "Fang (Equatorial Guinea)",
+        "fat": "Fanti",
+        "fao": "ferščina",
+        "fij": "fidžijščina",
+        "fil": "Filipino",
+        "fin": "finščina",
+        "fon": "Fon",
+        "fra": "francoščina",
+        "fur": "furlanščina",
+        "ful": "fulščina",
+        "gaa": "Ga",
+        "glg": "Galician",
+        "lug": "Ganda",
+        "gay": "gajščina?",
+        "gba": "Gbaya (Central African Republic)",
+        "hmj": "Ge",
+        "gez": "etiopščina?",
+        "kat": "gruzinščina",
+        "deu": "nemški",
+        "gil": "gilbertščina",
+        "gon": "Gondi",
+        "gor": "Gorontalo",
+        "got": "gotščina",
+        "grb": "Grebo",
+        "grn": "gvaranijščina",
+        "guj": "gudžaratščina",
+        "gwi": "Gwichʼin",
+        "hai": "haidščina",
+        "hau": "havščina",
+        "haw": "havajščina",
+        "heb": "hebrejščina",
+        "her": "Herero",
+        "hil": "hilingajnonščina",
+        "hin": "hindijščina",
+        "hmo": "hiri motu",
+        "hit": "hetitščina",
+        "hmn": "hmonščina; miaojščina",
+        "hun": "madžarščina",
+        "hup": "hupščina",
+        "iba": "ibanščina",
+        "isl": "islandščina",
+        "ido": "Ido",
+        "ibo": "Igbo",
+        "ilo": "Iloko",
+        "ind": "indonezijščina",
+        "inh": "inguščina",
+        "ina": "interlingva",
+        "ile": "Interlingue",
+        "iku": "inuktituščina",
+        "ipk": "Inupiaq",
+        "gle": "irščina",
+        "ita": "italijanščina",
+        "jpn": "japonščina",
+        "jav": "javanščina",
+        "jrb": "Judeo-Arabic",
+        "jpr": "Judeo-Persian",
+        "kbd": "kabardinščina",
+        "kab": "Kabyle",
+        "kac": "Kachin",
+        "kal": "Kalaallisut",
+        "xal": "Kalmyk",
+        "kam": "Kamba (Kenya)",
+        "kan": "kanareščina",
+        "kau": "Kanuri",
+        "kaa": "Kara-Kalpak",
+        "krc": "Karachay-Balkar",
+        "krl": "Karelian",
+        "kas": "kašmirščina",
+        "csb": "Kashubian",
+        "kaw": "kavi",
+        "kaz": "kazaščina",
+        "kha": "Khasi",
+        "kho": "Khotanese",
+        "kik": "kikujščina",
+        "kmb": "Kimbundu",
+        "kin": "Kinyarwanda",
+        "kir": "kirgiščina",
+        "tlh": "Klingon",
+        "kom": "komijščina",
+        "kon": "Kongo",
+        "kok": "Konkani (macrolanguage)",
+        "kor": "korejščina",
+        "kos": "Kosraean",
+        "kpe": "Kpelle",
+        "kua": "Kuanyama",
+        "kum": "kumiščina",
+        "kur": "kurdščina",
+        "kru": "Kurukh",
+        "kut": "kutenajščina",
+        "lad": "ladinščina",
+        "lah": "Lahnda",
+        "lam": "Lamba",
+        "lao": "laoščina",
+        "lat": "latinščina",
+        "lav": "latvijščina",
+        "lez": "lezginščina",
+        "lim": "Limburgan",
+        "lin": "lingala",
+        "lit": "litvanščina",
+        "jbo": "Lojban",
+        "loz": "Lozi",
+        "lub": "Luba-Katanga",
+        "lua": "lubalulujščina",
+        "lui": "Luiseno",
+        "smj": "Lule Sami",
+        "lun": "Lunda",
+        "luo": "Luo (Kenya and Tanzania)",
+        "lus": "Lushai",
+        "ltz": "Luxembourgish",
+        "mkd": "makedonščina",
+        "mad": "madurščina",
+        "mag": "Magahi",
+        "mai": "Maithili",
+        "mak": "makasarščina",
+        "mlg": "malgaščina",
+        "msa": "Malay (macrolanguage)",
+        "mal": "malajalščina",
+        "mlt": "malteščina",
+        "mnc": "Manchu",
+        "mdr": "Mandar",
+        "man": "Mandingo",
+        "mni": "manipurščina",
+        "glv": "manska gelščina",
+        "mri": "maorščina",
+        "arn": "Mapudungun",
+        "mar": "maratščina",
+        "chm": "Mari (Russia)",
+        "mah": "Marshallese",
+        "mwr": "Marwari",
+        "mas": "masajščina",
+        "men": "Mende (Sierra Leone)",
+        "mic": "Mi'kmaq",
+        "min": "Minangkabau",
+        "mwl": "Mirandese",
+        "moh": "mohoščina",
+        "mdf": "Moksha",
+        "lol": "Mongo",
+        "mon": "mongolščina",
+        "mos": "mosanščina",
+        "mul": "Več jezikov",
+        "nqo": "N'Ko",
+        "nau": "Nauru",
+        "nav": "navaščina",
+        "ndo": "Ndonga",
+        "nap": "napolitanščina",
+        "nia": "niaščina",
+        "niu": "niuejščina",
+        "zxx": "No linguistic content",
+        "nog": "Nogai",
+        "nor": "norveščina",
+        "nob": "Norwegian Bokmål",
+        "nno": "norveščina; nynorsk",
+        "nym": "Nyamwezi",
+        "nya": "Nyanja",
+        "nyn": "Nyankole",
+        "nyo": "Nyoro",
+        "nzi": "Nzima",
+        "oci": "Occitan (post 1500)",
+        "oji": "Ojibwa",
+        "orm": "Oromo",
+        "osa": "Osage",
+        "oss": "Ossetian",
+        "pal": "Pahlavi",
+        "pau": "palavanščina",
+        "pli": "Pali",
+        "pam": "Pampanga",
+        "pag": "pangasinanščina",
+        "pan": "Panjabi",
+        "pap": "papiamentu",
+        "fas": "perzijščina",
+        "phn": "feničanščina",
+        "pon": "Pohnpeian",
+        "pol": "poljščina",
+        "por": "portugalđščina",
+        "pus": "paštu",
+        "que": "Quechua",
+        "raj": "radžastanščina",
+        "rap": "rapanujščina",
+        "ron": "romunščina",
+        "roh": "Romansh",
+        "rom": "romščina",
+        "run": "rundščina",
+        "rus": "ruščina",
+        "smo": "samoanščina",
+        "sad": "Sandawe",
+        "sag": "Sango",
+        "san": "sanskrt",
+        "sat": "santalščina",
+        "srd": "sardinščina",
+        "sas": "Sasak",
+        "sco": "škotščina",
+        "sel": "selkupščina",
+        "srp": "srbščina",
+        "srr": "Serer",
+        "shn": "šanščina",
+        "sna": "šonščina",
+        "scn": "sicilijanščina",
+        "sid": "Sidamo",
+        "bla": "Siksika",
+        "snd": "sindščina",
+        "sin": "Sinhala",
+        "den": "Slave (Athapascan)",
+        "slk": "slovaščina",
+        "slv": "slovenščina",
+        "sog": "Sogdian",
+        "som": "Somali",
+        "snk": "Soninke",
+        "spa": "španščina",
+        "srn": "Sranan Tongo",
+        "suk": "Sukuma",
+        "sux": "sumerščina",
+        "sun": "sundščina",
+        "sus": "susuamijščina?",
+        "swa": "Swahili (macrolanguage)",
+        "ssw": "svazijščina?",
+        "swe": "švedščina",
+        "syr": "sirščina",
+        "tgl": "tagaloščina",
+        "tah": "tahitijščina",
+        "tgk": "tadžiščina",
+        "tmh": "Tamashek",
+        "tam": "tamilščina",
+        "tat": "tatarščina",
+        "tel": "Telugu",
+        "ter": "Tereno",
+        "tet": "Tetum",
+        "tha": "tajščina",
+        "bod": "tibetanščina",
+        "tig": "Tigre",
+        "tir": "Tigrinya",
+        "tem": "Timne",
+        "tiv": "Tiv",
+        "tli": "Tlingit",
+        "tpi": "tok pisin",
+        "tkl": "Tokelau",
+        "tog": "Tonga (Nyasa)",
+        "ton": "tonganščina",
+        "tsi": "tsimšijščina",
+        "tso": "Tsonga",
+        "tsn": "Tswana",
+        "tum": "Tumbuka",
+        "tur": "turščina",
+        "tuk": "turkmenščina",
+        "tvl": "tuvalujščina",
+        "tyv": "Tuvinian",
+        "twi": "Twi",
+        "udm": "Udmurt",
+        "uga": "ugaritščina",
+        "uig": "ujgurščina",
+        "ukr": "ukrajinščina",
+        "umb": "Umbundu",
+        "mis": "Uncoded languages",
+        "und": "nedoločen",
+        "urd": "urdujščina",
+        "uzb": "uzbeščina",
+        "vai": "vajščina",
+        "ven": "Venda",
+        "vie": "vietnamščina",
+        "vol": "Volapük",
+        "vot": "votjaščina",
+        "wln": "valonščina",
+        "war": "Waray (Philippines)",
+        "was": "Washo",
+        "cym": "valižanščina",
+        "wal": "Wolaytta",
+        "wol": "Wolof",
+        "xho": "koščina",
+        "sah": "jakutščina",
+        "yao": "jaojščina",
+        "yap": "Yapese",
+        "yid": "jidiš",
+        "yor": "jorubščina",
+        "zap": "Zapotec",
+        "zza": "Zaza",
+        "zen": "Zenaga",
+        "zha": "Zhuang",
+        "zul": "zulujščina",
+        "zun": "Zuni"
+    },
     "sv": {
         "aar": "Afar",
         "abk": "Abchaziska",

cps/jinjia.py

@@ -27,7 +27,7 @@ import datetime
 import mimetypes
 from uuid import uuid4
 
-from flask import Blueprint, request, url_for
+from flask import Blueprint, request, url_for, g
 from flask_babel import format_date
 from .cw_login import current_user
 
@@ -111,11 +111,15 @@ def yesno(value, yes, no):
 
 @jinjia.app_template_filter('formatfloat')
 def formatfloat(value, decimals=1):
-    value = 0 if not value else value
-    return ('{0:.' + str(decimals) + 'f}').format(value).rstrip('0').rstrip('.')
+    if not value:
+        return value
+    formated_value = ('{0:.' + str(decimals) + 'f}').format(value)
+    if formated_value.endswith('.' + "0" * decimals):
+        formated_value = formated_value.rstrip('0').rstrip('.')
+    return formated_value
 
 
-@jinjia.app_template_filter('formatseriesindex')
+'''@jinjia.app_template_filter('formatseriesindex')
 def formatseriesindex_filter(series_index):
     if series_index:
         try:
@@ -126,6 +130,7 @@ def formatseriesindex_filter(series_index):
         except (ValueError, TypeError):
             return series_index
     return 0
+'''
 
 
 @jinjia.app_template_filter('escapedlink')
@@ -179,3 +184,12 @@ def get_cover_srcset(series):
         url = url_for('web.get_series_cover', series_id=series.id, resolution=shortname, c=cache_timestamp())
         srcset.append(f'{url} {resolution}x')
     return ', '.join(srcset)
+
+
+@jinjia.app_template_filter('music')
+def contains_music(book_formats):
+    result = False
+    for format in book_formats:
+        if format.format.lower() in g.constants.EXTENSIONS_AUDIO:
+            result = True
+    return result

cps/kobo.py

@@ -18,7 +18,7 @@
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
 import base64
-import datetime
+from datetime import datetime, timezone
 import os
 import uuid
 import zipfile
@@ -47,7 +47,7 @@ import requests
 from . import config, logger, kobo_auth, db, calibre_db, helper, shelf as shelf_lib, ub, csrf, kobo_sync_status
 from . import isoLanguages
 from .epub import get_epub_layout
-from .constants import COVER_THUMBNAIL_SMALL
+from .constants import COVER_THUMBNAIL_SMALL, COVER_THUMBNAIL_MEDIUM, COVER_THUMBNAIL_LARGE
 from .helper import get_download_link
 from .services import SyncToken as SyncToken
 from .web import download_required
@@ -131,7 +131,7 @@ def convert_to_kobo_timestamp_string(timestamp):
         return timestamp.strftime("%Y-%m-%dT%H:%M:%SZ")
     except AttributeError as exc:
         log.debug("Timestamp not valid: {}".format(exc))
-        return datetime.datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
+        return datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
 
 
 @kobo.route("/v1/library/sync")
@@ -150,15 +150,15 @@ def HandleSyncRequest():
 
     # if no books synced don't respect sync_token
     if not ub.session.query(ub.KoboSyncedBooks).filter(ub.KoboSyncedBooks.user_id == current_user.id).count():
-        sync_token.books_last_modified = datetime.datetime.min
-        sync_token.books_last_created = datetime.datetime.min
-        sync_token.reading_state_last_modified = datetime.datetime.min
+        sync_token.books_last_modified = datetime.min
+        sync_token.books_last_created = datetime.min
+        sync_token.reading_state_last_modified = datetime.min
 
     new_books_last_modified = sync_token.books_last_modified  # needed for sync selected shelfs only
     new_books_last_created = sync_token.books_last_created  # needed to distinguish between new and changed entitlement
     new_reading_state_last_modified = sync_token.reading_state_last_modified
 
-    new_archived_last_modified = datetime.datetime.min
+    new_archived_last_modified = datetime.min
     sync_results = []
 
     # We reload the book database so that the user gets a fresh view of the library
@@ -375,7 +375,7 @@ def create_book_entitlement(book, archived):
     book_uuid = str(book.uuid)
     return {
         "Accessibility": "Full",
-        "ActivePeriod": {"From": convert_to_kobo_timestamp_string(datetime.datetime.utcnow())},
+        "ActivePeriod": {"From": convert_to_kobo_timestamp_string(datetime.now(timezone.utc))},
         "Created": convert_to_kobo_timestamp_string(book.timestamp),
         "CrossRevisionId": book_uuid,
         "Id": book_uuid,
@@ -423,7 +423,7 @@ def get_series(book):
 
 
 def get_seriesindex(book):
-    return book.series_index or 1
+    return book.series_index if isinstance(book.series_index, float) else 1
 
 
 def get_language(book):
@@ -486,14 +486,16 @@ def get_metadata(book):
 
     if get_series(book):
         name = get_series(book)
-        metadata["Series"] = {
-            "Name": get_series(book),
-            "Number": get_seriesindex(book),        # ToDo Check int() ?
-            "NumberFloat": float(get_seriesindex(book)),
-            # Get a deterministic id based on the series name.
-            "Id": str(uuid.uuid3(uuid.NAMESPACE_DNS, name)),
-        }
-
+        try:
+            metadata["Series"] = {
+                "Name": get_series(book),
+                "Number": get_seriesindex(book),        # ToDo Check int() ?
+                "NumberFloat": float(get_seriesindex(book)),
+                # Get a deterministic id based on the series name.
+                "Id": str(uuid.uuid3(uuid.NAMESPACE_DNS, name)),
+            }
+        except Exception as e:
+            print(e)
     return metadata
 
 
@@ -795,7 +797,7 @@ def HandleStateRequest(book_uuid):
                 if new_book_read_status == ub.ReadBook.STATUS_IN_PROGRESS \
                         and new_book_read_status != book_read.read_status:
                     book_read.times_started_reading += 1
-                    book_read.last_time_started_reading = datetime.datetime.utcnow()
+                    book_read.last_time_started_reading = datetime.now(timezone.utc)
                 book_read.read_status = new_book_read_status
                 update_results_response["StatusInfoResult"] = {"Result": "Success"}
         except (KeyError, TypeError, ValueError, StatementError):
@@ -903,7 +905,12 @@ def get_current_bookmark_response(current_bookmark):
 @requires_kobo_auth
 def HandleCoverImageRequest(book_uuid, width, height, Quality, isGreyscale):
     try:
-        resolution = None if int(height) > 1000 else COVER_THUMBNAIL_SMALL
+        if int(height) > 1000:
+            resolution = COVER_THUMBNAIL_LARGE
+        elif int(height) > 500:
+            resolution = COVER_THUMBNAIL_MEDIUM
+        else:
+            resolution = COVER_THUMBNAIL_SMALL
     except ValueError:
         log.error("Requested height %s of book %s is invalid" % (book_uuid, height))
         resolution = COVER_THUMBNAIL_SMALL

cps/kobo_sync_status.py

@@ -19,7 +19,7 @@
 
 from .cw_login import current_user
 from . import ub
-import datetime
+from datetime import datetime, timezone
 from sqlalchemy.sql.expression import or_, and_, true
 # from sqlalchemy import exc
 
@@ -58,7 +58,7 @@ def change_archived_books(book_id, state=None, message=None):
         archived_book = ub.ArchivedBook(user_id=current_user.id, book_id=book_id)
 
     archived_book.is_archived = state if state else not archived_book.is_archived
-    archived_book.last_modified = datetime.datetime.utcnow()        # toDo. Check utc timestamp
+    archived_book.last_modified = datetime.now(timezone.utc)        # toDo. Check utc timestamp
 
     ub.session.merge(archived_book)
     ub.session_commit(message)

cps/metadata_provider/amazon.py

@@ -38,14 +38,16 @@ class Amazon(Metadata):
     __name__ = "Amazon"
     __id__ = "amazon"
     headers = {'upgrade-insecure-requests': '1',
-               'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/97.0.4692.71 Safari/537.36',
-               'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9',
-               'sec-gpc': '1',
-               'sec-fetch-site': 'none',
-               'sec-fetch-mode': 'navigate',
-               'sec-fetch-user': '?1',
-               'sec-fetch-dest': 'document',
-               'accept-encoding': 'gzip, deflate, br',
+               'user-agent': 'Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0',
+               'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8',
+               'Sec-Fetch-Site': 'same-origin',
+               'Sec-Fetch-Mode': 'navigate',
+               'Sec-Fetch-User': '?1',
+               'Sec-Fetch-Dest': 'document',
+               'Upgrade-Insecure-Requests': '1',
+               'Alt-Used' : 'www.amazon.com',
+               'Priority' : 'u=0, i',
+               'accept-encoding': 'gzip, deflate, br, zstd',
                'accept-language': 'en-US,en;q=0.9'}
     session = requests.Session()
     session.headers=headers
@@ -53,7 +55,6 @@ class Amazon(Metadata):
     def search(
         self, query: str, generic_cover: str = "", locale: str = "en"
     ) -> Optional[List[MetaRecord]]:
-        #timer=time()
         def inner(link, index) -> [dict, int]:
             with self.session as session:
                 try:
@@ -61,11 +62,11 @@ class Amazon(Metadata):
                     r.raise_for_status()
                 except Exception as ex:
                     log.warning(ex)
-                    return None
+                    return []
                 long_soup = BS(r.text, "lxml")  #~4sec :/
-                soup2 = long_soup.find("div", attrs={"cel_widget_id": "dpx-books-ppd_csm_instrumentation_wrapper"})
+                soup2 = long_soup.find("div", attrs={"cel_widget_id": "dpx-ppd_csm_instrumentation_wrapper"})
                 if soup2 is None:
-                    return None
+                    return []
                 try:
                     match = MetaRecord(
                         title = "",
@@ -88,7 +89,7 @@ class Amazon(Metadata):
                             soup2.find("div", attrs={"data-feature-name": "bookDescription"}).stripped_strings)\
                                                 .replace("\xa0"," ")[:-9].strip().strip("\n")
                     except (AttributeError, TypeError):
-                        return None  # if there is no description it is not a book and therefore should be ignored
+                        return []  # if there is no description it is not a book and therefore should be ignored
                     try:
                         match.title = soup2.find("span", attrs={"id": "productTitle"}).text
                     except (AttributeError, TypeError):
@@ -107,13 +108,13 @@ class Amazon(Metadata):
                     except (AttributeError, ValueError):
                         match.rating = 0
                     try:
-                        match.cover = soup2.find("img", attrs={"class": "a-dynamic-image frontImage"})["src"]
+                        match.cover = soup2.find("img", attrs={"class": "a-dynamic-image"})["src"]
                     except (AttributeError, TypeError):
                         match.cover = ""
                     return match, index
                 except Exception as e:
                     log.error_or_exception(e)
-                    return None
+                    return []
 
         val = list()
         if self.active:
@@ -133,7 +134,7 @@ class Amazon(Metadata):
             links_list = [next(filter(lambda i: "digital-text" in i["href"], x.findAll("a")))["href"] for x in
                           soup.findAll("div", attrs={"data-component-type": "s-search-result"})]
             with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
-                fut = {executor.submit(inner, link, index) for index, link in enumerate(links_list[:5])}
-                val = list(map(lambda x : x.result() ,concurrent.futures.as_completed(fut)))
+                fut = {executor.submit(inner, link, index) for index, link in enumerate(links_list[:3])}
+                val = list(map(lambda x : x.result(), concurrent.futures.as_completed(fut)))
         result = list(filter(lambda x: x, val))
         return [x[0] for x in sorted(result, key=itemgetter(1))] #sort by amazons listing order for best relevance

cps/metadata_provider/google.py

@@ -54,7 +54,7 @@ class Google(Metadata):
                 results.raise_for_status()
             except Exception as e:
                 log.warning(e)
-                return None
+                return []
             for result in results.json().get("items", []):
                 val.append(
                     self._parse_search_result(

cps/opds.py

@@ -424,11 +424,8 @@ def feed_shelf(book_id):
 @requires_basic_auth_if_no_ano
 def opds_download_link(book_id, book_format):
     if not auth.current_user().role_download():
-        return abort(403)
-    if "Kobo" in request.headers.get('User-Agent'):
-        client = "kobo"
-    else:
-        client = ""
+        return abort(401)
+    client = "kobo" if "Kobo" in request.headers.get('User-Agent') else ""
     return get_download_link(book_id, book_format.lower(), client)
 
 

cps/search.py

@@ -24,9 +24,9 @@ from flask_babel import format_date
 from flask_babel import gettext as _
 from sqlalchemy.sql.expression import func, not_, and_, or_, text, true
 from sqlalchemy.sql.functions import coalesce
-from sqlalchemy import exists
 
 from . import logger, db, calibre_db, config, ub
+from .string_helper import strip_whitespaces
 from .usermanagement import login_required_if_no_ano
 from .render_template import render_title_template
 from .pagination import Pagination
@@ -244,7 +244,8 @@ def render_adv_search_results(term, offset=None, order=None, limit=None):
     pagination = None
 
     cc = calibre_db.get_cc_columns(config, filter_config_custom_read=True)
-    calibre_db.session.connection().connection.connection.create_function("lower", 1, db.lcase)
+    calibre_db.create_functions()
+    # calibre_db.session.connection().connection.connection.create_function("lower", 1, db.lcase)
     query = calibre_db.generate_linked_query(config.config_read_column, db.Books)
     q = query.outerjoin(db.books_series_link, db.Books.id == db.books_series_link.c.book)\
         .outerjoin(db.Series)\
@@ -257,21 +258,21 @@ def render_adv_search_results(term, offset=None, order=None, limit=None):
         tags['include_' + element] = term.get('include_' + element)
         tags['exclude_' + element] = term.get('exclude_' + element)
 
-    author_name = term.get("author_name")
-    book_title = term.get("book_title")
+    author_name = term.get("authors")
+    book_title = term.get("title")
     publisher = term.get("publisher")
     pub_start = term.get("publishstart")
     pub_end = term.get("publishend")
     rating_low = term.get("ratinghigh")
     rating_high = term.get("ratinglow")
-    description = term.get("comment")
+    description = term.get("comments")
     read_status = term.get("read_status")
     if author_name:
-        author_name = author_name.strip().lower().replace(',', '|')
+        author_name = strip_whitespaces(author_name).lower().replace(',', '|')
     if book_title:
-        book_title = book_title.strip().lower()
+        book_title = strip_whitespaces(book_title).lower()
     if publisher:
-        publisher = publisher.strip().lower()
+        publisher = strip_whitespaces(publisher).lower()
 
     search_term = []
     cc_present = False

cps/search_metadata.py

@@ -33,7 +33,6 @@ from cps.services.Metadata import Metadata
 from . import constants, logger, ub, web_server
 from .usermanagement import user_login_required
 
-# current_milli_time = lambda: int(round(time() * 1000))
 
 meta = Blueprint("metadata", __name__)
 
@@ -130,7 +129,7 @@ def metadata_search():
     locale = get_locale()
     if query:
         static_cover = url_for("static", filename="generic_cover.jpg")
-        # start = current_milli_time()
+        # ret = cl[0].search(query, static_cover, locale)
         with concurrent.futures.ThreadPoolExecutor(max_workers=5) as executor:
             meta = {
                 executor.submit(c.search, query, static_cover, locale): c
@@ -139,5 +138,4 @@ def metadata_search():
             }
             for future in concurrent.futures.as_completed(meta):
                 data.extend([asdict(x) for x in future.result() if x])
-    # log.info({'Time elapsed {}'.format(current_milli_time()-start)})
     return Response(json.dumps(data), mimetype="application/json")

cps/services/background_scheduler.py

@@ -42,6 +42,7 @@ class BackgroundScheduler:
         if cls._instance is None:
             cls._instance = super(BackgroundScheduler, cls).__new__(cls)
             cls.log = logger.create()
+            logger.logging.getLogger('tzlocal').setLevel(logger.logging.WARNING)
             cls.scheduler = BScheduler()
             cls.scheduler.start()
 

cps/shelf.py

@@ -21,7 +21,7 @@
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
 
 import sys
-from datetime import datetime
+from datetime import datetime, timezone
 
 from flask import Blueprint, flash, redirect, request, url_for, abort
 from flask_babel import gettext as _
@@ -80,7 +80,7 @@ def add_to_shelf(shelf_id, book_id):
         return "%s is a invalid Book Id. Could not be added to Shelf" % book_id, 400
 
     shelf.books.append(ub.BookShelf(shelf=shelf.id, book_id=book_id, order=maxOrder + 1))
-    shelf.last_modified = datetime.utcnow()
+    shelf.last_modified = datetime.now(timezone.utc)
     try:
         ub.session.merge(shelf)
         ub.session.commit()
@@ -139,7 +139,7 @@ def search_to_shelf(shelf_id):
         for book in books_for_shelf:
             maxOrder += 1
             shelf.books.append(ub.BookShelf(shelf=shelf.id, book_id=book, order=maxOrder))
-        shelf.last_modified = datetime.utcnow()
+        shelf.last_modified = datetime.now(timezone.utc)
         try:
             ub.session.merge(shelf)
             ub.session.commit()
@@ -185,7 +185,7 @@ def remove_from_shelf(shelf_id, book_id):
 
         try:
             ub.session.delete(book_shelf)
-            shelf.last_modified = datetime.utcnow()
+            shelf.last_modified = datetime.now(timezone.utc)
             ub.session.commit()
         except (OperationalError, InvalidRequestError) as e:
             ub.session.rollback()
@@ -250,7 +250,7 @@ def show_simpleshelf(shelf_id):
     return render_show_shelf(2, shelf_id, 1, None)
 
 
-@shelf.route("/shelf/<int:shelf_id>", defaults={"sort_param": "order", 'page': 1})
+@shelf.route("/shelf/<int:shelf_id>", defaults={"sort_param": "stored", 'page': 1})
 @shelf.route("/shelf/<int:shelf_id>/<sort_param>", defaults={'page': 1})
 @shelf.route("/shelf/<int:shelf_id>/<sort_param>/<int:page>")
 @login_required_if_no_ano
@@ -271,7 +271,7 @@ def order_shelf(shelf_id):
             for book in books_in_shelf:
                 setattr(book, 'order', to_save[str(book.book_id)])
                 counter += 1
-                # if order different from before -> shelf.last_modified = datetime.utcnow()
+                # if order different from before -> shelf.last_modified = datetime.now(timezone.utc)
             try:
                 ub.session.commit()
             except (OperationalError, InvalidRequestError) as e:
@@ -418,29 +418,37 @@ def change_shelf_order(shelf_id, order):
 
 def render_show_shelf(shelf_type, shelf_id, page_no, sort_param):
     shelf = ub.session.query(ub.Shelf).filter(ub.Shelf.id == shelf_id).first()
-
+    status = current_user.get_view_property("shelf", 'man')
     # check user is allowed to access shelf
     if shelf and check_shelf_view_permissions(shelf):
         if shelf_type == 1:
-            # order = [ub.BookShelf.order.asc()]
-            if sort_param == 'pubnew':
-                change_shelf_order(shelf_id, [db.Books.pubdate.desc()])
-            if sort_param == 'pubold':
-                change_shelf_order(shelf_id, [db.Books.pubdate])
-            if sort_param == 'abc':
-                change_shelf_order(shelf_id, [db.Books.sort])
-            if sort_param == 'zyx':
-                change_shelf_order(shelf_id, [db.Books.sort.desc()])
-            if sort_param == 'new':
-                change_shelf_order(shelf_id, [db.Books.timestamp.desc()])
-            if sort_param == 'old':
-                change_shelf_order(shelf_id, [db.Books.timestamp])
-            if sort_param == 'authaz':
-                change_shelf_order(shelf_id, [db.Books.author_sort.asc(), db.Series.name, db.Books.series_index])
-            if sort_param == 'authza':
-                change_shelf_order(shelf_id, [db.Books.author_sort.desc(),
-                                              db.Series.name.desc(),
-                                              db.Books.series_index.desc()])
+            if status != 'on':
+                if sort_param == 'stored':
+                    sort_param = current_user.get_view_property("shelf", 'stored')
+                else:
+                    current_user.set_view_property("shelf", 'stored', sort_param)
+                if sort_param == 'pubnew':
+                    change_shelf_order(shelf_id, [db.Books.pubdate.desc()])
+                if sort_param == 'pubold':
+                    change_shelf_order(shelf_id, [db.Books.pubdate])
+                if sort_param == 'shelfnew':
+                    change_shelf_order(shelf_id, [ub.BookShelf.date_added.desc()])
+                if sort_param == 'shelfold':
+                    change_shelf_order(shelf_id, [ub.BookShelf.date_added])
+                if sort_param == 'abc':
+                    change_shelf_order(shelf_id, [db.Books.sort])
+                if sort_param == 'zyx':
+                    change_shelf_order(shelf_id, [db.Books.sort.desc()])
+                if sort_param == 'new':
+                    change_shelf_order(shelf_id, [db.Books.timestamp.desc()])
+                if sort_param == 'old':
+                    change_shelf_order(shelf_id, [db.Books.timestamp])
+                if sort_param == 'authaz':
+                    change_shelf_order(shelf_id, [db.Books.author_sort.asc(), db.Series.name, db.Books.series_index])
+                if sort_param == 'authza':
+                    change_shelf_order(shelf_id, [db.Books.author_sort.desc(),
+                                                  db.Series.name.desc(),
+                                                  db.Books.series_index.desc()])
             page = "shelf.html"
             pagesize = 0
         else:
@@ -453,7 +461,7 @@ def render_show_shelf(shelf_type, shelf_id, page_no, sort_param):
                                                            [ub.BookShelf.order.asc()],
                                                            True, config.config_read_column,
                                                            ub.BookShelf, ub.BookShelf.book_id == db.Books.id)
-        # delete chelf entries where book is not existent anymore, can happen if book is deleted outside calibre-web
+        # delete shelf entries where book is not existent anymore, can happen if book is deleted outside calibre-web
         wrong_entries = calibre_db.session.query(ub.BookShelf) \
             .join(db.Books, ub.BookShelf.book_id == db.Books.id, isouter=True) \
             .filter(db.Books.id == None).all()
@@ -472,7 +480,9 @@ def render_show_shelf(shelf_type, shelf_id, page_no, sort_param):
                                      pagination=pagination,
                                      title=_("Shelf: '%(name)s'", name=shelf.name),
                                      shelf=shelf,
-                                     page="shelf")
+                                     page="shelf",
+                                     status=status,
+                                     order=sort_param)
     else:
         flash(_("Error opening shelf. Shelf does not exist or is not accessible"), category="error")
         return redirect(url_for("web.index"))

cps/static/css/caliBlur.css

@@ -3268,6 +3268,10 @@ div.btn-group[role=group][aria-label="Download, send to Kindle, reading"] > div.
     left: auto !important
 }
 
+ul.dropdown-menu.offscreen {
+    margin: 2px -160px 0
+}
+
 div.btn-group[role=group][aria-label="Download, send to Kindle, reading"] .dropdown-menu.offscreen {
     position: fixed;
     top: 120px;
@@ -4333,6 +4337,7 @@ body.advanced_search > div.container-fluid > div.row-fluid > div.col-sm-10 > div
 
 .navbar-right > li > ul.dropdown-menu.offscreen {
     right: -10px
+
 }
 
 .login .plexBack, body.login > div.container-fluid > div.row-fluid > div.col-sm-2, body.login > div.navbar.navbar-default.navbar-static-top > div > form {
@@ -7951,3 +7956,5 @@ div.comments[data-readmore] {
     transition: height 300ms;
     overflow: hidden
 }
+
+.dropdown-menu > .offscreen

cps/static/css/epub_themes.css

@@ -15,5 +15,5 @@
 
 .blackTheme {
 	background: #000;
-	color: #fff
+	color: #fff;
 }

cps/static/css/main.css

@@ -77,7 +77,6 @@ body {
 }
 
 #panels a {
-  visibility: hidden;
   width: 18px;
   height: 20px;
   overflow: hidden;

cps/static/js/edit_books.js

@@ -3,9 +3,9 @@
  */
 /* global Bloodhound, language, Modernizr, tinymce, getPath */
 
-if ($("#description").length) {
+if ($("#comments").length && typeof tinymce !== "undefined") {
     tinymce.init({
-        selector: "#description",
+        selector: "#comments",
         plugins: 'code',
         branding: false,
         menubar: "edit view format",
@@ -93,7 +93,7 @@ var authors = new Bloodhound({
     },
 });
 
-$(".form-group #bookAuthor").typeahead(
+$(".form-group #authors").typeahead(
     {
         highlight: true,
         minLength: 1,
@@ -243,13 +243,13 @@ $("#search").on("change input.typeahead:selected", function(event) {
     });
 });
 
-$("#btn-upload-format").on("change", function () {
+/*$("#btn-upload-format").on("change", function () {
     var filename = $(this).val();
     if (filename.substring(3, 11) === "fakepath") {
         filename = filename.substring(12);
     } // Remove c:\fake at beginning from localhost chrome
     $("#upload-format").text(filename);
-});
+});*/
 
 $("#btn-upload-cover").on("change", function () {
     var filename = $(this).val();
@@ -261,8 +261,8 @@ $("#btn-upload-cover").on("change", function () {
 
 $("#xchange").click(function () {
     this.blur();
-    var title = $("#book_title").val();
-    $("#book_title").val($("#bookAuthor").val());
-    $("#bookAuthor").val(title);
+    var title = $("#title").val();
+    $("#title").val($("#authors").val());
+    $("#authors").val(title);
 });
 

cps/static/js/get_meta.js

@@ -38,12 +38,12 @@ $(function () {
     }
 
     function populateForm (book) {
-        tinymce.get("description").setContent(book.description);
+        tinymce.get("comments").setContent(book.description);
         var uniqueTags = getUniqueValues('tags', book)
         var uniqueLanguages = getUniqueValues('languages', book)
         var ampSeparatedAuthors = (book.authors || []).join(" & ");
-        $("#bookAuthor").val(ampSeparatedAuthors);
-        $("#book_title").val(book.title);
+        $("#authors").val(ampSeparatedAuthors);
+        $("#title").val(book.title);
         $("#tags").val(uniqueTags.join(", "));
         $("#languages").val(uniqueLanguages.join(", "));
         $("#rating").data("rating").setValue(Math.round(book.rating));
@@ -172,7 +172,7 @@ $(function () {
 
     $("#get_meta").click(function () {
         populate_provider();
-        var bookTitle = $("#book_title").val();
+        var bookTitle = $("#title").val();
         $("#keyword").val(bookTitle);
         keyword = bookTitle;
         doSearch(bookTitle);

cps/static/js/libs/bootstrap-datepicker/locales/bootstrap-datepicker.sl.min.js

@@ -0,0 +1 @@
+!function(a){a.fn.datepicker.dates.sl={days:["Nedelja","Ponedeljek","Torek","Sreda","Četrtek","Petek","Sobota"],daysShort:["Ned","Pon","Tor","Sre","Čet","Pet","Sob"],daysMin:["Ne","Po","To","Sr","Če","Pe","So"],months:["Januar","Februar","Marec","April","Maj","Junij","Julij","Avgust","September","Oktober","November","December"],monthsShort:["Jan","Feb","Mar","Apr","Maj","Jun","Jul","Avg","Sep","Okt","Nov","Dec"],today:"Danes",weekStart:1}}(jQuery);

cps/static/js/libs/viewer.mjs

@@ -2,7 +2,7 @@
  * @licstart The following is the entire license notice for the
  * JavaScript code in this page
  *
- * Copyright 2023 Mozilla Foundation
+ * Copyright 2024 Mozilla Foundation
  *
  * Licensed under the Apache License, Version 2.0 (the "License");
  * you may not use this file except in compliance with the License.
@@ -446,7 +446,7 @@ class ProgressBar {
     }
   }
   setDisableAutoFetch(delay = 5000) {
-    if (isNaN(this.#percent)) {
+    if (this.#percent === 100 || isNaN(this.#percent)) {
       return;
     }
     if (this.#disableAutoFetchTimeout) {
@@ -535,15 +535,20 @@ function toggleExpandedBtn(button, toggle, view = null) {
 
 ;// CONCATENATED MODULE: ./web/app_options.js
 {
-  var compatibilityParams = Object.create(null);
+  var compatParams = new Map();
   const userAgent = navigator.userAgent || "";
   const platform = navigator.platform || "";
   const maxTouchPoints = navigator.maxTouchPoints || 1;
   const isAndroid = /Android/.test(userAgent);
   const isIOS = /\b(iPad|iPhone|iPod)(?=;)/.test(userAgent) || platform === "MacIntel" && maxTouchPoints > 1;
-  (function checkCanvasSizeLimitation() {
+  (function () {
     if (isIOS || isAndroid) {
-      compatibilityParams.maxCanvasPixels = 5242880;
+      compatParams.set("maxCanvasPixels", 5242880);
+    }
+  })();
+  (function () {
+    if (isAndroid) {
+      compatParams.set("useSystemFonts", false);
     }
   })();
 }
@@ -552,9 +557,21 @@ const OptionKind = {
   VIEWER: 0x02,
   API: 0x04,
   WORKER: 0x08,
+  EVENT_DISPATCH: 0x10,
   PREFERENCE: 0x80
 };
+const Type = {
+  BOOLEAN: 0x01,
+  NUMBER: 0x02,
+  OBJECT: 0x04,
+  STRING: 0x08,
+  UNDEFINED: 0x10
+};
 const defaultOptions = {
+  allowedGlobalEvents: {
+    value: null,
+    kind: OptionKind.BROWSER
+  },
   canvasMaxAreaInBytes: {
     value: -1,
     kind: OptionKind.BROWSER + OptionKind.API
@@ -563,6 +580,16 @@ const defaultOptions = {
     value: false,
     kind: OptionKind.BROWSER
   },
+  localeProperties: {
+    value: {
+      lang: navigator.language || "en-US"
+    },
+    kind: OptionKind.BROWSER
+  },
+  nimbusDataStr: {
+    value: "",
+    kind: OptionKind.BROWSER
+  },
   supportsCaretBrowsingMode: {
     value: false,
     kind: OptionKind.BROWSER
@@ -587,6 +614,14 @@ const defaultOptions = {
     value: true,
     kind: OptionKind.BROWSER
   },
+  toolbarDensity: {
+    value: 0,
+    kind: OptionKind.BROWSER + OptionKind.EVENT_DISPATCH
+  },
+  altTextLearnMoreUrl: {
+    value: "",
+    kind: OptionKind.VIEWER + OptionKind.PREFERENCE
+  },
   annotationEditorMode: {
     value: 0,
     kind: OptionKind.VIEWER + OptionKind.PREFERENCE
@@ -619,6 +654,14 @@ const defaultOptions = {
     value: false,
     kind: OptionKind.VIEWER + OptionKind.PREFERENCE
   },
+  enableAltText: {
+    value: false,
+    kind: OptionKind.VIEWER + OptionKind.PREFERENCE
+  },
+  enableGuessAltText: {
+    value: true,
+    kind: OptionKind.VIEWER + OptionKind.PREFERENCE
+  },
   enableHighlightEditor: {
     value: false,
     kind: OptionKind.VIEWER + OptionKind.PREFERENCE
@@ -627,10 +670,6 @@ const defaultOptions = {
     value: false,
     kind: OptionKind.VIEWER + OptionKind.PREFERENCE
   },
-  enableML: {
-    value: false,
-    kind: OptionKind.VIEWER + OptionKind.PREFERENCE
-  },
   enablePermissions: {
     value: false,
     kind: OptionKind.VIEWER + OptionKind.PREFERENCE
@@ -643,8 +682,8 @@ const defaultOptions = {
     value: true,
     kind: OptionKind.VIEWER + OptionKind.PREFERENCE
   },
-  enableStampEditor: {
-    value: true,
+  enableUpdatedAddImage: {
+    value: false,
     kind: OptionKind.VIEWER + OptionKind.PREFERENCE
   },
   externalLinkRel: {
@@ -775,6 +814,11 @@ const defaultOptions = {
     value: "../web/standard_fonts/",
     kind: OptionKind.API
   },
+  useSystemFonts: {
+    value: undefined,
+    kind: OptionKind.API,
+    type: Type.BOOLEAN + Type.UNDEFINED
+  },
   verbosity: {
     value: 1,
     kind: OptionKind.API
@@ -807,62 +851,80 @@ const defaultOptions = {
     value: false,
     kind: OptionKind.VIEWER
   };
-  defaultOptions.locale = {
-    value: navigator.language || "en-US",
-    kind: OptionKind.VIEWER
-  };
 }
-const userOptions = Object.create(null);
+const userOptions = new Map();
 {
-  for (const name in compatibilityParams) {
-    userOptions[name] = compatibilityParams[name];
+  for (const [name, value] of compatParams) {
+    userOptions.set(name, value);
   }
 }
 class AppOptions {
+  static eventBus;
   constructor() {
     throw new Error("Cannot initialize AppOptions.");
   }
   static get(name) {
-    return userOptions[name] ?? defaultOptions[name]?.value ?? undefined;
+    return userOptions.has(name) ? userOptions.get(name) : defaultOptions[name]?.value;
   }
   static getAll(kind = null, defaultOnly = false) {
     const options = Object.create(null);
     for (const name in defaultOptions) {
-      const defaultOption = defaultOptions[name];
-      if (kind && !(kind & defaultOption.kind)) {
+      const defaultOpt = defaultOptions[name];
+      if (kind && !(kind & defaultOpt.kind)) {
         continue;
       }
-      options[name] = defaultOnly ? defaultOption.value : userOptions[name] ?? defaultOption.value;
+      options[name] = !defaultOnly && userOptions.has(name) ? userOptions.get(name) : defaultOpt.value;
     }
     return options;
   }
   static set(name, value) {
-    userOptions[name] = value;
+    this.setAll({
+      [name]: value
+    });
   }
-  static setAll(options, init = false) {
-    if (init) {
-      if (this.get("disablePreferences")) {
-        return;
-      }
-      for (const name in userOptions) {
-        if (compatibilityParams[name] !== undefined) {
-          continue;
-        }
-        console.warn("setAll: The Preferences may override manually set AppOptions; " + 'please use the "disablePreferences"-option in order to prevent that.');
-        break;
-      }
-    }
+  static setAll(options, prefs = false) {
+    let events;
     for (const name in options) {
-      userOptions[name] = options[name];
+      const defaultOpt = defaultOptions[name],
+        userOpt = options[name];
+      if (!defaultOpt || !(typeof userOpt === typeof defaultOpt.value || Type[(typeof userOpt).toUpperCase()] & defaultOpt.type)) {
+        continue;
+      }
+      const {
+        kind
+      } = defaultOpt;
+      if (prefs && !(kind & OptionKind.BROWSER || kind & OptionKind.PREFERENCE)) {
+        continue;
+      }
+      if (this.eventBus && kind & OptionKind.EVENT_DISPATCH) {
+        (events ||= new Map()).set(name, userOpt);
+      }
+      userOptions.set(name, userOpt);
+    }
+    if (events) {
+      for (const [name, value] of events) {
+        this.eventBus.dispatch(name.toLowerCase(), {
+          source: this,
+          value
+        });
+      }
     }
   }
-  static remove(name) {
-    delete userOptions[name];
-    const val = compatibilityParams[name];
-    if (val !== undefined) {
-      userOptions[name] = val;
+}
+{
+  AppOptions._checkDisablePreferences = () => {
+    if (AppOptions.get("disablePreferences")) {
+      return true;
     }
-  }
+    for (const [name] of userOptions) {
+      if (compatParams.has(name)) {
+        continue;
+      }
+      console.warn("The Preferences may override manually set AppOptions; " + 'please use the "disablePreferences"-option to prevent that.');
+      break;
+    }
+    return false;
+  };
 }
 
 ;// CONCATENATED MODULE: ./web/pdf_link_service.js
@@ -1171,26 +1233,27 @@ class PDFLinkService {
     if (!(typeof zoom === "object" && typeof zoom?.name === "string")) {
       return false;
     }
+    const argsLen = args.length;
     let allowNull = true;
     switch (zoom.name) {
       case "XYZ":
-        if (args.length !== 3) {
+        if (argsLen < 2 || argsLen > 3) {
           return false;
         }
         break;
       case "Fit":
       case "FitB":
-        return args.length === 0;
+        return argsLen === 0;
       case "FitH":
       case "FitBH":
       case "FitV":
       case "FitBV":
-        if (args.length !== 1) {
+        if (argsLen > 1) {
           return false;
         }
         break;
       case "FitR":
-        if (args.length !== 4) {
+        if (argsLen !== 4) {
           return false;
         }
         allowNull = false;
@@ -1240,7 +1303,6 @@ const {
   noContextMenu,
   normalizeUnicode,
   OPS,
-  Outliner,
   PasswordResponses,
   PDFDataRangeTransport,
   PDFDateString,
@@ -1248,12 +1310,10 @@ const {
   PermissionFlag,
   PixelsPerInch,
   RenderingCancelledException,
-  renderTextLayer,
   setLayerDimensions,
   shadow,
   TextLayer,
   UnexpectedResponseException,
-  updateTextLayer,
   Util,
   VerbosityLevel,
   version,
@@ -1401,40 +1461,28 @@ class BaseExternalServices {
   updateEditorStates(data) {
     throw new Error("Not implemented: updateEditorStates");
   }
-  async getNimbusExperimentData() {}
-  async getGlobalEventNames() {
-    return null;
-  }
   dispatchGlobalEvent(_event) {}
 }
 
 ;// CONCATENATED MODULE: ./web/preferences.js
 
 class BasePreferences {
-  #browserDefaults = Object.freeze({
-    canvasMaxAreaInBytes: -1,
-    isInAutomation: false,
-    supportsCaretBrowsingMode: false,
-    supportsDocumentFonts: true,
-    supportsIntegratedFind: false,
-    supportsMouseWheelZoomCtrlKey: true,
-    supportsMouseWheelZoomMetaKey: true,
-    supportsPinchToZoom: true
-  });
   #defaults = Object.freeze({
+    altTextLearnMoreUrl: "",
     annotationEditorMode: 0,
     annotationMode: 2,
     cursorToolOnLoad: 0,
     defaultZoomDelay: 400,
     defaultZoomValue: "",
     disablePageLabels: false,
+    enableAltText: false,
+    enableGuessAltText: true,
     enableHighlightEditor: false,
     enableHighlightFloatingButton: false,
-    enableML: false,
     enablePermissions: false,
     enablePrintAutoRotate: true,
     enableScripting: true,
-    enableStampEditor: true,
+    enableUpdatedAddImage: false,
     externalLinkTarget: 0,
     highlightEditorColors: "yellow=#FFFF98,green=#53FFBC,blue=#80EBFF,pink=#FFCBE6,red=#FF4F5F",
     historyUpdateUrl: false,
@@ -1456,7 +1504,6 @@ class BasePreferences {
     enableXfa: true,
     viewerCssTheme: 0
   });
-  #prefs = Object.create(null);
   #initializedPromise = null;
   constructor() {
     if (this.constructor === BasePreferences) {
@@ -1466,16 +1513,13 @@ class BasePreferences {
       browserPrefs,
       prefs
     }) => {
-      const options = Object.create(null);
-      for (const [name, val] of Object.entries(this.#browserDefaults)) {
-        const prefVal = browserPrefs?.[name];
-        options[name] = typeof prefVal === typeof val ? prefVal : val;
+      if (AppOptions._checkDisablePreferences()) {
+        return;
       }
-      for (const [name, val] of Object.entries(this.#defaults)) {
-        const prefVal = prefs?.[name];
-        options[name] = this.#prefs[name] = typeof prefVal === typeof val ? prefVal : val;
-      }
-      AppOptions.setAll(options, true);
+      AppOptions.setAll({
+        ...browserPrefs,
+        ...prefs
+      }, true);
     });
   }
   async _writeToStorage(prefObj) {
@@ -1484,58 +1528,21 @@ class BasePreferences {
   async _readFromStorage(prefObj) {
     throw new Error("Not implemented: _readFromStorage");
   }
-  #updatePref({
-    name,
-    value
-  }) {
-    throw new Error("Not implemented: #updatePref");
-  }
   async reset() {
     await this.#initializedPromise;
-    const oldPrefs = structuredClone(this.#prefs);
-    this.#prefs = Object.create(null);
-    try {
-      await this._writeToStorage(this.#defaults);
-    } catch (reason) {
-      this.#prefs = oldPrefs;
-      throw reason;
-    }
+    AppOptions.setAll(this.#defaults, true);
+    await this._writeToStorage(this.#defaults);
   }
   async set(name, value) {
     await this.#initializedPromise;
-    const defaultValue = this.#defaults[name],
-      oldPrefs = structuredClone(this.#prefs);
-    if (defaultValue === undefined) {
-      throw new Error(`Set preference: "${name}" is undefined.`);
-    } else if (value === undefined) {
-      throw new Error("Set preference: no value is specified.");
-    }
-    const valueType = typeof value,
-      defaultType = typeof defaultValue;
-    if (valueType !== defaultType) {
-      if (valueType === "number" && defaultType === "string") {
-        value = value.toString();
-      } else {
-        throw new Error(`Set preference: "${value}" is a ${valueType}, expected a ${defaultType}.`);
-      }
-    } else if (valueType === "number" && !Number.isInteger(value)) {
-      throw new Error(`Set preference: "${value}" must be an integer.`);
-    }
-    this.#prefs[name] = value;
-    try {
-      await this._writeToStorage(this.#prefs);
-    } catch (reason) {
-      this.#prefs = oldPrefs;
-      throw reason;
-    }
+    AppOptions.setAll({
+      [name]: value
+    }, true);
+    await this._writeToStorage(AppOptions.getAll(OptionKind.PREFERENCE));
   }
   async get(name) {
     await this.#initializedPromise;
-    const defaultValue = this.#defaults[name];
-    if (defaultValue === undefined) {
-      throw new Error(`Get preference: "${name}" is undefined.`);
-    }
-    return this.#prefs[name] ?? defaultValue;
+    return AppOptions.get(name);
   }
   get initializedPromise() {
     return this.#initializedPromise;
@@ -3098,13 +3105,19 @@ class Preferences extends BasePreferences {
 }
 class ExternalServices extends BaseExternalServices {
   async createL10n() {
-    return new genericl10n_GenericL10n(AppOptions.get("locale"));
+    return new genericl10n_GenericL10n(AppOptions.get("localeProperties")?.lang);
   }
   createScripting() {
     return new GenericScripting(AppOptions.get("sandboxBundleSrc"));
   }
 }
 class MLManager {
+  async isEnabledFor(_name) {
+    return false;
+  }
+  async deleteModel(_service) {
+    return null;
+  }
   async guess() {
     return null;
   }
@@ -8411,6 +8424,9 @@ class AnnotationLayerBuilder {
     }
     this.div.hidden = true;
   }
+  hasEditableAnnotations() {
+    return !!this.annotationLayer?.hasEditableAnnotations();
+  }
   #updatePresentationModeState(state) {
     if (!this.div) {
       return;
@@ -9142,6 +9158,7 @@ class PDFPageView {
   #annotationMode = AnnotationMode.ENABLE_FORMS;
   #enableHWA = false;
   #hasRestrictedScaling = false;
+  #isEditing = false;
   #layerProperties = null;
   #loadingId = null;
   #previousRotation = null;
@@ -9296,6 +9313,9 @@ class PDFPageView {
     this.reset();
     this.pdfPage?.cleanup();
   }
+  hasEditableAnnotations() {
+    return !!this.annotationLayer?.hasEditableAnnotations();
+  }
   get _textHighlighter() {
     return shadow(this, "_textHighlighter", new TextHighlighter({
       pageIndex: this.id - 1,
@@ -9472,6 +9492,19 @@ class PDFPageView {
       this._resetZoomLayer();
     }
   }
+  toggleEditingMode(isEditing) {
+    if (!this.hasEditableAnnotations()) {
+      return;
+    }
+    this.#isEditing = isEditing;
+    this.reset({
+      keepZoomLayer: true,
+      keepAnnotationLayer: true,
+      keepAnnotationEditorLayer: true,
+      keepXfaLayer: true,
+      keepTextLayer: true
+    });
+  }
   update({
     scale = 0,
     rotation = null,
@@ -9822,7 +9855,8 @@ class PDFPageView {
       annotationMode: this.#annotationMode,
       optionalContentConfigPromise: this._optionalContentConfigPromise,
       annotationCanvasMap: this._annotationCanvasMap,
-      pageColors
+      pageColors,
+      isEditing: this.#isEditing
     };
     const renderTask = this.renderTask = pdfPage.render(renderContext);
     renderTask.onContinue = renderContinueCallback;
@@ -9982,8 +10016,11 @@ class PDFViewer {
   #enableHWA = false;
   #enableHighlightFloatingButton = false;
   #enablePermissions = false;
+  #enableUpdatedAddImage = false;
   #eventAbortController = null;
   #mlManager = null;
+  #onPageRenderedCallback = null;
+  #switchAnnotationEditorModeTimeoutId = null;
   #getAllTextInProgress = false;
   #hiddenCopyElement = null;
   #interruptCopyCondition = false;
@@ -9993,7 +10030,7 @@ class PDFViewer {
   #scaleTimeoutId = null;
   #textLayerMode = TextLayerMode.ENABLE;
   constructor(options) {
-    const viewerVersion = "4.4.168";
+    const viewerVersion = "4.5.136";
     if (version !== viewerVersion) {
       throw new Error(`The API version "${version}" does not match the Viewer version "${viewerVersion}".`);
     }
@@ -10020,6 +10057,7 @@ class PDFViewer {
     this.#annotationEditorMode = options.annotationEditorMode ?? AnnotationEditorType.NONE;
     this.#annotationEditorHighlightColors = options.annotationEditorHighlightColors || null;
     this.#enableHighlightFloatingButton = options.enableHighlightFloatingButton === true;
+    this.#enableUpdatedAddImage = options.enableUpdatedAddImage === true;
     this.imageResourcesPath = options.imageResourcesPath || "";
     this.enablePrintAutoRotate = options.enablePrintAutoRotate || false;
     this.removePageBorders = options.removePageBorders || false;
@@ -10425,7 +10463,7 @@ class PDFViewer {
         if (pdfDocument.isPureXfa) {
           console.warn("Warning: XFA-editing is not implemented.");
         } else if (isValidAnnotationEditorMode(mode)) {
-          this.#annotationEditorUIManager = new AnnotationEditorUIManager(this.container, viewer, this.#altTextManager, eventBus, pdfDocument, pageColors, this.#annotationEditorHighlightColors, this.#enableHighlightFloatingButton, this.#mlManager);
+          this.#annotationEditorUIManager = new AnnotationEditorUIManager(this.container, viewer, this.#altTextManager, eventBus, pdfDocument, pageColors, this.#annotationEditorHighlightColors, this.#enableHighlightFloatingButton, this.#enableUpdatedAddImage, this.#mlManager);
           eventBus.dispatch("annotationeditoruimanager", {
             source: this,
             uiManager: this.#annotationEditorUIManager
@@ -10584,6 +10622,7 @@ class PDFViewer {
     this.viewer.removeAttribute("lang");
     this.#hiddenCopyElement?.remove();
     this.#hiddenCopyElement = null;
+    this.#cleanupSwitchAnnotationEditorMode();
   }
   #ensurePageViewVisible() {
     if (this._scrollMode !== ScrollMode.PAGE) {
@@ -10956,6 +10995,34 @@ class PDFViewer {
       location: this._location
     });
   }
+  #switchToEditAnnotationMode() {
+    const visible = this._getVisiblePages();
+    const pagesToRefresh = [];
+    const {
+      ids,
+      views
+    } = visible;
+    for (const page of views) {
+      const {
+        view
+      } = page;
+      if (!view.hasEditableAnnotations()) {
+        ids.delete(view.id);
+        continue;
+      }
+      pagesToRefresh.push(page);
+    }
+    if (pagesToRefresh.length === 0) {
+      return null;
+    }
+    this.renderingQueue.renderHighestPriority({
+      first: pagesToRefresh[0],
+      last: pagesToRefresh.at(-1),
+      views: pagesToRefresh,
+      ids
+    });
+    return ids;
+  }
   containsElement(element) {
     return this.container.contains(element);
   }
@@ -11388,6 +11455,16 @@ class PDFViewer {
   get containerTopLeft() {
     return this.#containerTopLeft ||= [this.container.offsetTop, this.container.offsetLeft];
   }
+  #cleanupSwitchAnnotationEditorMode() {
+    if (this.#onPageRenderedCallback) {
+      this.eventBus._off("pagerendered", this.#onPageRenderedCallback);
+      this.#onPageRenderedCallback = null;
+    }
+    if (this.#switchAnnotationEditorModeTimeoutId !== null) {
+      clearTimeout(this.#switchAnnotationEditorModeTimeoutId);
+      this.#switchAnnotationEditorModeTimeoutId = null;
+    }
+  }
   get annotationEditorMode() {
     return this.#annotationEditorUIManager ? this.#annotationEditorMode : AnnotationEditorType.DISABLE;
   }
@@ -11408,12 +11485,47 @@ class PDFViewer {
     if (!this.pdfDocument) {
       return;
     }
-    this.#annotationEditorMode = mode;
-    this.eventBus.dispatch("annotationeditormodechanged", {
-      source: this,
-      mode
-    });
-    this.#annotationEditorUIManager.updateMode(mode, editId, isFromKeyboard);
+    const {
+      eventBus
+    } = this;
+    const updater = () => {
+      this.#cleanupSwitchAnnotationEditorMode();
+      this.#annotationEditorMode = mode;
+      this.#annotationEditorUIManager.updateMode(mode, editId, isFromKeyboard);
+      eventBus.dispatch("annotationeditormodechanged", {
+        source: this,
+        mode
+      });
+    };
+    if (mode === AnnotationEditorType.NONE || this.#annotationEditorMode === AnnotationEditorType.NONE) {
+      const isEditing = mode !== AnnotationEditorType.NONE;
+      if (!isEditing) {
+        this.pdfDocument.annotationStorage.resetModifiedIds();
+      }
+      for (const pageView of this._pages) {
+        pageView.toggleEditingMode(isEditing);
+      }
+      const idsToRefresh = this.#switchToEditAnnotationMode();
+      if (isEditing && idsToRefresh) {
+        this.#cleanupSwitchAnnotationEditorMode();
+        this.#onPageRenderedCallback = ({
+          pageNumber
+        }) => {
+          idsToRefresh.delete(pageNumber);
+          if (idsToRefresh.size === 0) {
+            this.#switchAnnotationEditorModeTimeoutId = setTimeout(updater, 0);
+          }
+        };
+        const {
+          signal
+        } = this.#eventAbortController;
+        eventBus._on("pagerendered", this.#onPageRenderedCallback, {
+          signal
+        });
+        return;
+      }
+    }
+    updater();
   }
   set annotationEditorParams({
     type,
@@ -11721,7 +11833,7 @@ class SecondaryToolbar {
 
 class Toolbar {
   #opts;
-  constructor(options, eventBus) {
+  constructor(options, eventBus, toolbarDensity = 0) {
     this.#opts = options;
     this.eventBus = eventBus;
     const buttons = [{
@@ -11806,8 +11918,13 @@ class Toolbar {
           break;
       }
     });
+    eventBus._on("toolbardensity", this.#updateToolbarDensity.bind(this));
+    this.#updateToolbarDensity({
+      value: toolbarDensity
+    });
     this.reset();
   }
+  #updateToolbarDensity() {}
   #setAnnotationEditorUIManager(uiManager, parentContainer) {
     const colorPicker = new ColorPicker({
       uiManager
@@ -12078,7 +12195,6 @@ class ViewHistory {
 
 
 const FORCE_PAGES_LOADED_TIMEOUT = 10000;
-const WHEEL_ZOOM_DISABLED_TIMEOUT = 1000;
 const ViewOnLoad = {
   UNKNOWN: -1,
   PREVIOUS: 0,
@@ -12110,18 +12226,17 @@ const PDFViewerApplication = {
   store: null,
   downloadManager: null,
   overlayManager: null,
-  preferences: null,
+  preferences: new Preferences(),
   toolbar: null,
   secondaryToolbar: null,
   eventBus: null,
   l10n: null,
   annotationEditorParams: null,
   isInitialViewSet: false,
-  downloadComplete: false,
   isViewerEmbedded: window.parent !== window,
   url: "",
   baseUrl: "",
-  _allowedGlobalEventsPromise: null,
+  mlManager: null,
   _downloadUrl: "",
   _eventBusAbortController: null,
   _windowAbortController: null,
@@ -12141,11 +12256,9 @@ const PDFViewerApplication = {
   _printAnnotationStoragePromise: null,
   _touchInfo: null,
   _isCtrlKeyDown: false,
-  _nimbusDataPromise: null,
   _caretBrowsing: null,
   _isScrolling: false,
   async initialize(appConfig) {
-    let l10nPromise;
     this.appConfig = appConfig;
     try {
       await this.preferences.initializedPromise;
@@ -12167,8 +12280,7 @@ const PDFViewerApplication = {
     if (mode) {
       document.documentElement.classList.add(mode);
     }
-    l10nPromise = this.externalServices.createL10n();
-    this.l10n = await l10nPromise;
+    this.l10n = await this.externalServices.createL10n();
     document.getElementsByTagName("html")[0].dir = this.l10n.getDirection();
     this.l10n.translate(appConfig.appContainer || document.documentElement);
     if (this.isViewerEmbedded && AppOptions.get("externalLinkTarget") === LinkTarget.NONE) {
@@ -12257,7 +12369,9 @@ const PDFViewerApplication = {
       }
     }
     if (params.has("locale")) {
-      AppOptions.set("locale", params.get("locale"));
+      AppOptions.set("localeProperties", {
+        lang: params.get("locale")
+      });
     }
   },
   async _initializeViewerComponents() {
@@ -12318,6 +12432,7 @@ const PDFViewerApplication = {
       annotationEditorMode,
       annotationEditorHighlightColors: AppOptions.get("highlightEditorColors"),
       enableHighlightFloatingButton: AppOptions.get("enableHighlightFloatingButton"),
+      enableUpdatedAddImage: AppOptions.get("enableUpdatedAddImage"),
       imageResourcesPath: AppOptions.get("imageResourcesPath"),
       enablePrintAutoRotate: AppOptions.get("enablePrintAutoRotate"),
       maxCanvasPixels: AppOptions.get("maxCanvasPixels"),
@@ -12355,9 +12470,6 @@ const PDFViewerApplication = {
     }
     if (appConfig.annotationEditorParams) {
       if (annotationEditorMode !== AnnotationEditorType.DISABLE) {
-        if (AppOptions.get("enableStampEditor")) {
-          appConfig.toolbar?.editorStampButton?.classList.remove("hidden");
-        }
         const editorHighlightButton = appConfig.toolbar?.editorHighlightButton;
         if (editorHighlightButton && AppOptions.get("enableHighlightEditor")) {
           editorHighlightButton.hidden = false;
@@ -12380,7 +12492,7 @@ const PDFViewerApplication = {
       });
     }
     if (appConfig.toolbar) {
-      this.toolbar = new Toolbar(appConfig.toolbar, eventBus);
+      this.toolbar = new Toolbar(appConfig.toolbar, eventBus, AppOptions.get("toolbarDensity"));
     }
     if (appConfig.secondaryToolbar) {
       this.secondaryToolbar = new SecondaryToolbar(appConfig.secondaryToolbar, eventBus);
@@ -12437,7 +12549,6 @@ const PDFViewerApplication = {
     }
   },
   async run(config) {
-    this.preferences = new Preferences();
     await this.initialize(config);
     const {
       appConfig,
@@ -12514,9 +12625,6 @@ const PDFViewerApplication = {
   get externalServices() {
     return shadow(this, "externalServices", new ExternalServices());
   },
-  get mlManager() {
-    return shadow(this, "mlManager", AppOptions.get("enableML") === true ? new MLManager() : null);
-  },
   get initialized() {
     return this._initializedCapability.settled;
   },
@@ -12597,12 +12705,10 @@ const PDFViewerApplication = {
     let title = pdfjs_getPdfFilenameFromUrl(url, "");
     if (!title) {
       try {
-        title = decodeURIComponent(getFilenameFromUrl(url)) || url;
-      } catch {
-        title = url;
-      }
+        title = decodeURIComponent(getFilenameFromUrl(url));
+      } catch {}
     }
-    this.setTitle(title);
+    this.setTitle(title || url);
   },
   setTitle(title = this._title) {
     this._title = title;
@@ -12648,7 +12754,6 @@ const PDFViewerApplication = {
     this.pdfLinkService.externalLinkEnabled = true;
     this.store = null;
     this.isInitialViewSet = false;
-    this.downloadComplete = false;
     this.url = "";
     this.baseUrl = "";
     this._downloadUrl = "";
@@ -12724,9 +12829,7 @@ const PDFViewerApplication = {
   async download(options = {}) {
     let data;
     try {
-      if (this.downloadComplete) {
-        data = await this.pdfDocument.getData();
-      }
+      data = await this.pdfDocument.getData();
     } catch {}
     this.downloadManager.download(data, this._downloadUrl, this._docFilename, options);
   },
@@ -12793,11 +12896,8 @@ const PDFViewerApplication = {
     return message;
   },
   progress(level) {
-    if (!this.loadingBar || this.downloadComplete) {
-      return;
-    }
     const percent = Math.round(level * 100);
-    if (percent <= this.loadingBar.percent) {
+    if (!this.loadingBar || percent <= this.loadingBar.percent) {
       return;
     }
     this.loadingBar.percent = percent;
@@ -12811,7 +12911,6 @@ const PDFViewerApplication = {
       length
     }) => {
       this._contentLength = length;
-      this.downloadComplete = true;
       this.loadingBar?.hide();
       firstPagePromise.then(() => {
         this.eventBus.dispatch("documentloaded", {
@@ -13413,9 +13512,6 @@ const PDFViewerApplication = {
       });
     }
     addWindowResolutionChange();
-    window.addEventListener("visibilitychange", webViewerVisibilityChange, {
-      signal
-    });
     window.addEventListener("wheel", webViewerWheel, {
       passive: false,
       signal
@@ -13730,7 +13826,7 @@ function webViewerHashchange(evt) {
   }
 }
 {
-  /*var webViewerFileInputChange = function (evt) {
+  var webViewerFileInputChange = function (evt) {
     if (PDFViewerApplication.pdfViewer?.isInPresentationMode) {
       return;
     }
@@ -13742,7 +13838,7 @@ function webViewerHashchange(evt) {
   };
   var webViewerOpenFile = function (evt) {
     PDFViewerApplication._openFileInput?.click();
-  };*/
+  };
 }
 function webViewerPresentationMode() {
   PDFViewerApplication.requestPresentationMode();
@@ -13876,20 +13972,6 @@ function webViewerPageChanging({
 function webViewerResolutionChange(evt) {
   PDFViewerApplication.pdfViewer.refresh();
 }
-function webViewerVisibilityChange(evt) {
-  if (document.visibilityState === "visible") {
-    setZoomDisabledTimeout();
-  }
-}
-let zoomDisabledTimeout = null;
-function setZoomDisabledTimeout() {
-  if (zoomDisabledTimeout) {
-    clearTimeout(zoomDisabledTimeout);
-  }
-  zoomDisabledTimeout = setTimeout(function () {
-    zoomDisabledTimeout = null;
-  }, WHEEL_ZOOM_DISABLED_TIMEOUT);
-}
 function webViewerWheel(evt) {
   const {
     pdfViewer,
@@ -13907,7 +13989,7 @@ function webViewerWheel(evt) {
   const origin = [evt.clientX, evt.clientY];
   if (isPinchToZoom || evt.ctrlKey && supportsMouseWheelZoomCtrlKey || evt.metaKey && supportsMouseWheelZoomMetaKey) {
     evt.preventDefault();
-    if (PDFViewerApplication._isScrolling || zoomDisabledTimeout || document.visibilityState === "hidden" || PDFViewerApplication.overlayManager.active) {
+    if (PDFViewerApplication._isScrolling || document.visibilityState === "hidden" || PDFViewerApplication.overlayManager.active) {
       return;
     }
     if (isPinchToZoom && supportsPinchToZoom) {
@@ -14335,14 +14417,20 @@ function webViewerReportTelemetry({
 }) {
   PDFViewerApplication.externalServices.reportTelemetry(details);
 }
+function webViewerSetPreference({
+  name,
+  value
+}) {
+  PDFViewerApplication.preferences.set(name, value);
+}
 
 ;// CONCATENATED MODULE: ./web/viewer.js
 
 
 
 
-const pdfjsVersion = "4.4.168";
-const pdfjsBuild = "19fbc8998";
+const pdfjsVersion = "4.5.136";
+const pdfjsBuild = "3a21f03b0";
 const AppConstants = {
   LinkTarget: LinkTarget,
   RenderingStates: RenderingStates,

cps/static/js/main.js

@@ -130,8 +130,13 @@ $(".container-fluid").bind('drop', function (e) {
             }
         });
         if (dt.files.length) {
-            $("#btn-upload")[0].files = dt.files;
-            $("#form-upload").submit();
+            if($("#btn-upload-format").length) {
+                $("#btn-upload-format")[0].files = dt.files;
+                $("#form-upload-format").submit();
+            } else {
+                $("#btn-upload")[0].files = dt.files;
+                $("#form-upload").submit();
+            }
         }
     }
 });
@@ -140,12 +145,25 @@ $("#btn-upload").change(function() {
     $("#form-upload").submit();
 });
 
+$("#btn-upload-format").change(function() {
+    $("#form-upload-format").submit();
+});
+
+
 $("#form-upload").uploadprogress({
-    redirect_url: getPath() + "/", //"{{ url_for('web.index')}}",
-    uploadedMsg: $("#form-upload").data("message"), //"{{_('Upload done, processing, please wait...')}}",
-    modalTitle: $("#form-upload").data("title"), //"{{_('Uploading...')}}",
-    modalFooter: $("#form-upload").data("footer"), //"{{_('Close')}}",
-    modalTitleFailed: $("#form-upload").data("failed") //"{{_('Error')}}"
+    redirect_url: getPath() + "/",
+    uploadedMsg: $("#form-upload").data("message"),
+    modalTitle: $("#form-upload").data("title"),
+    modalFooter: $("#form-upload").data("footer"),
+    modalTitleFailed: $("#form-upload").data("failed")
+});
+
+$("#form-upload-format").uploadprogress({
+    redirect_url: getPath() + "/",
+    uploadedMsg: $("#form-upload-format").data("message"),
+    modalTitle: $("#form-upload-format").data("title"),
+    modalFooter: $("#form-upload-format").data("footer"),
+    modalTitleFailed: $("#form-upload-format").data("failed")
 });
 
 $(document).ready(function() {
@@ -604,6 +622,7 @@ $(function() {
     });
 
     $("#toggle_order_shelf").click(function() {
+        $("#toggle_order_shelf").toggleClass("dummy");
         $("#new").toggleClass("disabled");
         $("#old").toggleClass("disabled");
         $("#asc").toggleClass("disabled");
@@ -612,9 +631,20 @@ $(function() {
         $("#auth_za").toggleClass("disabled");
         $("#pub_new").toggleClass("disabled");
         $("#pub_old").toggleClass("disabled");
+        $("#shelf_new").toggleClass("disabled");
+        $("#shelf_old").toggleClass("disabled");
         var alternative_text = $("#toggle_order_shelf").data('alt-text');
+        var status = $("#toggle_order_shelf").hasClass("dummy") ? "on" : "off";
         $("#toggle_order_shelf").data('alt-text', $("#toggle_order_shelf").html());
         $("#toggle_order_shelf").html(alternative_text);
+
+        $.ajax({
+            method:"post",
+            contentType: "application/json; charset=utf-8",
+            dataType: "json",
+            url: getPath() + "/ajax/view",
+            data: "{\"shelf\": {\"man\": \"" + status + "\"}}",
+        });
     });
 
     $("#btndeluser").click(function() {

cps/static/js/table.js

@@ -685,6 +685,17 @@ function ratingFormatter(value, row) {
     return (value/2);
 }
 
+function seriesIndexFormatter(value, row) {
+    if (!value) {
+        return value;
+    }
+    formated_value = Number(value).toFixed(2);
+    if (formated_value.endsWith(".00")) {
+        formated_value = parseInt(formated_value).toString();
+    }
+    return formated_value;
+}
+
 
 /* Do some hiding disabling after user list is loaded */
 function loadSuccess() {
@@ -849,6 +860,7 @@ function BookCheckboxChange(checkbox, userId, field) {
         },
         success: handleListServerResponse
     });
+    console.log("test");
 }
 
 

cps/string_helper.py

@@ -1,8 +1,7 @@
-#!/usr/bin/env python
 # -*- coding: utf-8 -*-
 
 #  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
-#    Copyright (C) 2019 decentral1se
+#    Copyright (C) 2024 OzzieIsaacs
 #
 #  This program is free software: you can redistribute it and/or modify
 #  it under the terms of the GNU General Public License as published by
@@ -16,28 +15,9 @@
 #
 #  You should have received a copy of the GNU General Public License
 #  along with this program. If not, see <http://www.gnu.org/licenses/>.
-#
-#  """Calibre-web distribution package setuptools installer."""
-
-from setuptools import setup
-import os
 import re
-import codecs
 
-here = os.path.abspath(os.path.dirname(__file__))
 
-def read(*parts):
-    with codecs.open(os.path.join(here, *parts), 'r') as fp:
-        return fp.read()
+def strip_whitespaces(text):
+    return re.sub(r"(^[\s\u200B-\u200D\ufeff]+)|([\s\u200B-\u200D\ufeff]+$)","", text)
 
-def find_version(*file_paths):
-    version_file = read(*file_paths)
-    version_match = re.search(r"^STABLE_VERSION\s+=\s+{['\"]version['\"]:\s*['\"](.*)['\"]}",
-                              version_file, re.M)
-    if version_match:
-        return version_match.group(1)
-    raise RuntimeError("Unable to find version string.")
-
-setup(
-    version=find_version("src", "calibreweb", "cps", "constants.py")
-)

cps/tasks/convert.py

@@ -1,339 +1,351 @@
-# -*- coding: utf-8 -*-
-
-#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
-#    Copyright (C) 2020 pwr
-#
-#  This program is free software: you can redistribute it and/or modify
-#  it under the terms of the GNU General Public License as published by
-#  the Free Software Foundation, either version 3 of the License, or
-#  (at your option) any later version.
-#
-#  This program is distributed in the hope that it will be useful,
-#  but WITHOUT ANY WARRANTY; without even the implied warranty of
-#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-#  GNU General Public License for more details.
-#
-#  You should have received a copy of the GNU General Public License
-#  along with this program. If not, see <http://www.gnu.org/licenses/>.
-
-import os
-import re
-from glob import glob
-from shutil import copyfile, copyfileobj
-from markupsafe import escape
-from time import time
-from uuid import uuid4
-
-from sqlalchemy.exc import SQLAlchemyError
-from flask_babel import lazy_gettext as N_
-
-from cps.services.worker import CalibreTask
-from cps import db
-from cps import logger, config
-from cps.subproc_wrapper import process_open
-from flask_babel import gettext as _
-from cps.kobo_sync_status import remove_synced_book
-from cps.ub import init_db_thread
-from cps.file_helper import get_temp_dir
-
-from cps.tasks.mail import TaskEmail
-from cps import gdriveutils, helper
-from cps.constants import SUPPORTED_CALIBRE_BINARIES
-
-log = logger.create()
-
-current_milli_time = lambda: int(round(time() * 1000))
-
-
-class TaskConvert(CalibreTask):
-    def __init__(self, file_path, book_id, task_message, settings, ereader_mail, user=None):
-        super(TaskConvert, self).__init__(task_message)
-        self.worker_thread = None
-        self.file_path = file_path
-        self.book_id = book_id
-        self.title = ""
-        self.settings = settings
-        self.ereader_mail = ereader_mail
-        self.user = user
-
-        self.results = dict()
-
-    def run(self, worker_thread):
-        self.worker_thread = worker_thread
-        if config.config_use_google_drive:
-            worker_db = db.CalibreDB(expire_on_commit=False, init=True)
-            cur_book = worker_db.get_book(self.book_id)
-            self.title = cur_book.title
-            data = worker_db.get_book_format(self.book_id, self.settings['old_book_format'])
-            df = gdriveutils.getFileFromEbooksFolder(cur_book.path,
-                                                     data.name + "." + self.settings['old_book_format'].lower())
-            df_cover = gdriveutils.getFileFromEbooksFolder(cur_book.path, "cover.jpg")
-            if df:
-                datafile_cover = None
-                datafile = os.path.join(config.get_book_path(),
-                                        cur_book.path,
-                                        data.name + "." + self.settings['old_book_format'].lower())
-                if df_cover:
-                    datafile_cover = os.path.join(config.get_book_path(),
-                                                  cur_book.path, "cover.jpg")
-                if not os.path.exists(os.path.join(config.get_book_path(), cur_book.path)):
-                    os.makedirs(os.path.join(config.get_book_path(), cur_book.path))
-                df.GetContentFile(datafile)
-                if df_cover:
-                    df_cover.GetContentFile(datafile_cover)
-                worker_db.session.close()
-            else:
-                # ToDo Include cover in error handling
-                error_message = _("%(format)s not found on Google Drive: %(fn)s",
-                                  format=self.settings['old_book_format'],
-                                  fn=data.name + "." + self.settings['old_book_format'].lower())
-                worker_db.session.close()
-                return self._handleError(error_message)
-
-        filename = self._convert_ebook_format()
-        if config.config_use_google_drive:
-            os.remove(self.file_path + '.' + self.settings['old_book_format'].lower())
-            if df_cover:
-                os.remove(os.path.join(config.config_calibre_dir, cur_book.path, "cover.jpg"))
-
-        if filename:
-            if config.config_use_google_drive:
-                # Upload files to gdrive
-                gdriveutils.updateGdriveCalibreFromLocal()
-                self._handleSuccess()
-            if self.ereader_mail:
-                # if we're sending to E-Reader after converting, create a one-off task and run it immediately
-                # todo: figure out how to incorporate this into the progress
-                try:
-                    EmailText = N_(u"%(book)s send to E-Reader", book=escape(self.title))                    
-                    for email in self.ereader_mail.split(','):
-                        email = email.strip()
-                        worker_thread.add(self.user, TaskEmail(self.settings['subject'],
-                                                               self.results["path"],
-                                                               filename,
-                                                               self.settings,
-                                                               email,
-                                                               EmailText,
-                                                               self.settings['body'],
-                                                               id=self.book_id,
-                                                               internal=True)
-                                          )
-                except Exception as ex:
-                    return self._handleError(str(ex))
-
-    def _convert_ebook_format(self):
-        error_message = None
-        local_db = db.CalibreDB(expire_on_commit=False, init=True)
-        file_path = self.file_path
-        book_id = self.book_id
-        format_old_ext = '.' + self.settings['old_book_format'].lower()
-        format_new_ext = '.' + self.settings['new_book_format'].lower()
-
-        # check to see if destination format already exists - or if book is in database
-        # if it does - mark the conversion task as complete and return a success
-        # this will allow to send to E-Reader workflow to continue to work
-        if os.path.isfile(file_path + format_new_ext) or\
-                local_db.get_book_format(self.book_id, self.settings['new_book_format']):
-            log.info("Book id %d already converted to %s", book_id, format_new_ext)
-            cur_book = local_db.get_book(book_id)
-            self.title = cur_book.title
-            self.results['path'] = cur_book.path
-            self.results['title'] = self.title
-            new_format = local_db.session.query(db.Data).filter(db.Data.book == book_id)\
-                .filter(db.Data.format == self.settings['new_book_format'].upper()).one_or_none()
-            if not new_format:
-                new_format = db.Data(name=os.path.basename(file_path),
-                                     book_format=self.settings['new_book_format'].upper(),
-                                     book=book_id, uncompressed_size=os.path.getsize(file_path + format_new_ext))
-                try:
-                    local_db.session.merge(new_format)
-                    local_db.session.commit()
-                except SQLAlchemyError as e:
-                    local_db.session.rollback()
-                    log.error("Database error: %s", e)
-                    local_db.session.close()
-                    self._handleError(N_("Oops! Database Error: %(error)s.", error=e))
-                    return
-                self._handleSuccess()
-                local_db.session.close()
-                return os.path.basename(file_path + format_new_ext)
-        else:
-            log.info("Book id %d - target format of %s does not exist. Moving forward with convert.",
-                     book_id,
-                     format_new_ext)
-
-        if config.config_kepubifypath and format_old_ext == '.epub' and format_new_ext == '.kepub':
-            check, error_message = self._convert_kepubify(file_path,
-                                                          format_old_ext,
-                                                          format_new_ext)
-        else:
-            # check if calibre converter-executable is existing
-            if not os.path.exists(config.config_converterpath):
-                self._handleError(N_("Calibre ebook-convert %(tool)s not found", tool=config.config_converterpath))
-                return
-            has_cover = local_db.get_book(book_id).has_cover
-            check, error_message = self._convert_calibre(file_path, format_old_ext, format_new_ext, has_cover)
-
-        if check == 0:
-            cur_book = local_db.get_book(book_id)
-            if os.path.isfile(file_path + format_new_ext):
-                new_format = local_db.session.query(db.Data).filter(db.Data.book == book_id) \
-                    .filter(db.Data.format == self.settings['new_book_format'].upper()).one_or_none()
-                if not new_format:
-                    new_format = db.Data(name=cur_book.data[0].name,
-                                         book_format=self.settings['new_book_format'].upper(),
-                                         book=book_id, uncompressed_size=os.path.getsize(file_path + format_new_ext))
-                    try:
-                        local_db.session.merge(new_format)
-                        local_db.session.commit()
-                        if self.settings['new_book_format'].upper() in ['KEPUB', 'EPUB', 'EPUB3']:
-                            ub_session = init_db_thread()
-                            remove_synced_book(book_id, True, ub_session)
-                            ub_session.close()
-                    except SQLAlchemyError as e:
-                        local_db.session.rollback()
-                        log.error("Database error: %s", e)
-                        local_db.session.close()
-                        self._handleError(error_message)
-                        return
-                self.results['path'] = cur_book.path
-                self.title = cur_book.title
-                self.results['title'] = self.title
-                if not config.config_use_google_drive:
-                    self._handleSuccess()
-                return os.path.basename(file_path + format_new_ext)
-            else:
-                error_message = N_('%(format)s format not found on disk', format=format_new_ext.upper())
-        local_db.session.close()
-        log.info("ebook converter failed with error while converting book")
-        if not error_message:
-            error_message = N_('Ebook converter failed with unknown error')
-        else:
-            log.error(error_message)
-        self._handleError(error_message)
-        return
-
-    def _convert_kepubify(self, file_path, format_old_ext, format_new_ext):
-        if config.config_embed_metadata and config.config_binariesdir:
-            tmp_dir, temp_file_name = helper.do_calibre_export(self.book_id, format_old_ext[1:])
-            filename = os.path.join(tmp_dir, temp_file_name + format_old_ext)
-            temp_file_path = tmp_dir
-        else:
-            filename = file_path + format_old_ext
-            temp_file_path = os.path.dirname(file_path)
-        quotes = [1, 3]
-        command = [config.config_kepubifypath, filename, '-o', temp_file_path, '-i']
-        try:
-            p = process_open(command, quotes)
-        except OSError as e:
-            return 1, N_("Kepubify-converter failed: %(error)s", error=e)
-        self.progress = 0.01
-        while True:
-            nextline = p.stdout.readlines()
-            nextline = [x.strip('\n') for x in nextline if x != '\n']
-            for line in nextline:
-                log.debug(line)
-            if p.poll() is not None:
-                break
-
-        # process returncode
-        check = p.returncode
-
-        # move file
-        if check == 0:
-            converted_file = glob(os.path.splitext(filename)[0] + "*.kepub.epub")
-            if len(converted_file) == 1:
-                copyfile(converted_file[0], (file_path + format_new_ext))
-                os.unlink(converted_file[0])
-            else:
-                return 1, N_("Converted file not found or more than one file in folder %(folder)s",
-                             folder=os.path.dirname(file_path))
-        return check, None
-
-    def _convert_calibre(self, file_path, format_old_ext, format_new_ext, has_cover):
-        path_tmp_opf = None
-        try:
-            # path_tmp_opf = self._embed_metadata()
-            if config.config_embed_metadata:
-                quotes = [5]
-                tmp_dir = get_temp_dir()
-                calibredb_binarypath = os.path.join(config.config_binariesdir, SUPPORTED_CALIBRE_BINARIES["calibredb"])
-                my_env = os.environ.copy()
-                if config.config_calibre_split:
-                    my_env['CALIBRE_OVERRIDE_DATABASE_PATH'] = os.path.join(config.config_calibre_dir, "metadata.db")
-                    library_path = config.config_calibre_split_dir
-                else:
-                    library_path = config.config_calibre_dir
-
-                opf_command = [calibredb_binarypath, 'show_metadata', '--as-opf', str(self.book_id),
-                               '--with-library', library_path]
-                p = process_open(opf_command, quotes, my_env)
-                p.wait()
-                check = p.returncode
-                calibre_traceback = p.stderr.readlines()
-                if check == 0:
-                    path_tmp_opf = os.path.join(tmp_dir, "metadata_" + str(uuid4()) + ".opf")
-                    with open(path_tmp_opf, 'w') as fd:
-                        copyfileobj(p.stdout, fd)
-                else:
-                    error_message = ""
-                    for ele in calibre_traceback:
-                        if not ele.startswith('Traceback') and not ele.startswith('  File'):
-                            error_message = N_("Calibre failed with error: %(error)s", error=ele)
-                    return check, error_message
-            quotes = [1, 2, 4, 6]
-            command = [config.config_converterpath, (file_path + format_old_ext),
-                       (file_path + format_new_ext)]
-            if config.config_embed_metadata:
-                command.extend(['--from-opf', path_tmp_opf])
-            if has_cover:
-                command.extend(['--cover', os.path.join(os.path.dirname(file_path), 'cover.jpg')])
-            quotes_index = 3
-            if config.config_calibre:
-                parameters = config.config_calibre.split(" ")
-                for param in parameters:
-                    command.append(param)
-                    quotes.append(quotes_index)
-                    quotes_index += 1
-
-            p = process_open(command, quotes, newlines=False)
-        except OSError as e:
-            return 1, N_("Ebook-converter failed: %(error)s", error=e)
-
-        while p.poll() is None:
-            nextline = p.stdout.readline()
-            if isinstance(nextline, bytes):
-                nextline = nextline.decode('utf-8', errors="ignore").strip('\r\n')
-            if nextline:
-                log.debug(nextline)
-            # parse progress string from calibre-converter
-            progress = re.search(r"(\d+)%\s.*", nextline)
-            if progress:
-                self.progress = int(progress.group(1)) / 100
-                if config.config_use_google_drive:
-                    self.progress *= 0.9
-
-        # process returncode
-        check = p.returncode
-        calibre_traceback = p.stderr.readlines()
-        error_message = ""
-        for ele in calibre_traceback:
-            ele = ele.decode('utf-8', errors="ignore").strip('\n')
-            log.debug(ele)
-            if not ele.startswith('Traceback') and not ele.startswith('  File'):
-                error_message = N_("Calibre failed with error: %(error)s", error=ele)
-        return check, error_message
-
-    @property
-    def name(self):
-        return N_("Convert")
-
-    def __str__(self):
-        if self.ereader_mail:
-            return "Convert Book {} and mail it to {}".format(self.book_id, self.ereader_mail)
-        else:
-            return "Convert Book {}".format(self.book_id)
-
-    @property
-    def is_cancellable(self):
-        return False
+# -*- coding: utf-8 -*-
+
+#  This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+#    Copyright (C) 2020 pwr
+#
+#  This program is free software: you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation, either version 3 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program. If not, see <http://www.gnu.org/licenses/>.
+
+import os
+import re
+import glob
+from shutil import copyfile, copyfileobj
+from markupsafe import escape
+from time import time
+from uuid import uuid4
+
+from sqlalchemy.exc import SQLAlchemyError
+from flask_babel import lazy_gettext as N_
+
+from cps.services.worker import CalibreTask
+from cps import db
+from cps import logger, config
+from cps.subproc_wrapper import process_open
+from flask_babel import gettext as _
+from cps.kobo_sync_status import remove_synced_book
+from cps.ub import init_db_thread
+from cps.file_helper import get_temp_dir
+
+from cps.tasks.mail import TaskEmail
+from cps import gdriveutils, helper
+from cps.constants import SUPPORTED_CALIBRE_BINARIES
+from cps.string_helper import strip_whitespaces
+
+log = logger.create()
+
+current_milli_time = lambda: int(round(time() * 1000))
+
+
+class TaskConvert(CalibreTask):
+    def __init__(self, file_path, book_id, task_message, settings, ereader_mail, user=None):
+        super(TaskConvert, self).__init__(task_message)
+        self.worker_thread = None
+        self.file_path = file_path
+        self.book_id = book_id
+        self.title = ""
+        self.settings = settings
+        self.ereader_mail = ereader_mail
+        self.user = user
+
+        self.results = dict()
+
+    def run(self, worker_thread):
+        self.worker_thread = worker_thread
+        if config.config_use_google_drive:
+            worker_db = db.CalibreDB(expire_on_commit=False, init=True)
+            cur_book = worker_db.get_book(self.book_id)
+            self.title = cur_book.title
+            data = worker_db.get_book_format(self.book_id, self.settings['old_book_format'])
+            df = gdriveutils.getFileFromEbooksFolder(cur_book.path,
+                                                     data.name + "." + self.settings['old_book_format'].lower())
+            df_cover = gdriveutils.getFileFromEbooksFolder(cur_book.path, "cover.jpg")
+            if df:
+                datafile_cover = None
+                datafile = os.path.join(config.get_book_path(),
+                                        cur_book.path,
+                                        data.name + "." + self.settings['old_book_format'].lower())
+                if df_cover:
+                    datafile_cover = os.path.join(config.get_book_path(),
+                                                  cur_book.path, "cover.jpg")
+                if not os.path.exists(os.path.join(config.get_book_path(), cur_book.path)):
+                    os.makedirs(os.path.join(config.get_book_path(), cur_book.path))
+                df.GetContentFile(datafile)
+                if df_cover:
+                    df_cover.GetContentFile(datafile_cover)
+                worker_db.session.close()
+            else:
+                # ToDo Include cover in error handling
+                error_message = _("%(format)s not found on Google Drive: %(fn)s",
+                                  format=self.settings['old_book_format'],
+                                  fn=data.name + "." + self.settings['old_book_format'].lower())
+                worker_db.session.close()
+                return self._handleError(error_message)
+
+        filename = self._convert_ebook_format()
+        if config.config_use_google_drive:
+            os.remove(self.file_path + '.' + self.settings['old_book_format'].lower())
+            if df_cover:
+                os.remove(os.path.join(config.config_calibre_dir, cur_book.path, "cover.jpg"))
+
+        if filename:
+            if config.config_use_google_drive:
+                # Upload files to gdrive
+                gdriveutils.updateGdriveCalibreFromLocal()
+                self._handleSuccess()
+            if self.ereader_mail:
+                # if we're sending to E-Reader after converting, create a one-off task and run it immediately
+                # todo: figure out how to incorporate this into the progress
+                try:
+                    EmailText = N_(u"%(book)s send to E-Reader", book=escape(self.title))                    
+                    for email in self.ereader_mail.split(','):
+                        email = strip_whitespaces(email)
+                        worker_thread.add(self.user, TaskEmail(self.settings['subject'],
+                                                               self.results["path"],
+                                                               filename,
+                                                               self.settings,
+                                                               email,
+                                                               EmailText,
+                                                               self.settings['body'],
+                                                               id=self.book_id,
+                                                               internal=True)
+                                          )
+                except Exception as ex:
+                    return self._handleError(str(ex))
+
+    def _convert_ebook_format(self):
+        error_message = None
+        local_db = db.CalibreDB(expire_on_commit=False, init=True)
+        file_path = self.file_path
+        book_id = self.book_id
+        format_old_ext = '.' + self.settings['old_book_format'].lower()
+        format_new_ext = '.' + self.settings['new_book_format'].lower()
+
+        # check to see if destination format already exists - or if book is in database
+        # if it does - mark the conversion task as complete and return a success
+        # this will allow to send to E-Reader workflow to continue to work
+        if os.path.isfile(file_path + format_new_ext) or\
+                local_db.get_book_format(self.book_id, self.settings['new_book_format']):
+            log.info("Book id %d already converted to %s", book_id, format_new_ext)
+            cur_book = local_db.get_book(book_id)
+            self.title = cur_book.title
+            self.results['path'] = cur_book.path
+            self.results['title'] = self.title
+            new_format = local_db.session.query(db.Data).filter(db.Data.book == book_id)\
+                .filter(db.Data.format == self.settings['new_book_format'].upper()).one_or_none()
+            if not new_format:
+                new_format = db.Data(name=os.path.basename(file_path),
+                                     book_format=self.settings['new_book_format'].upper(),
+                                     book=book_id, uncompressed_size=os.path.getsize(file_path + format_new_ext))
+                try:
+                    local_db.session.merge(new_format)
+                    local_db.session.commit()
+                except SQLAlchemyError as e:
+                    local_db.session.rollback()
+                    log.error("Database error: %s", e)
+                    local_db.session.close()
+                    self._handleError(N_("Oops! Database Error: %(error)s.", error=e))
+                    return
+                self._handleSuccess()
+                local_db.session.close()
+                return os.path.basename(file_path + format_new_ext)
+        else:
+            log.info("Book id %d - target format of %s does not exist. Moving forward with convert.",
+                     book_id,
+                     format_new_ext)
+
+        if config.config_kepubifypath and format_old_ext == '.epub' and format_new_ext == '.kepub':
+            check, error_message = self._convert_kepubify(file_path,
+                                                          format_old_ext,
+                                                          format_new_ext)
+        else:
+            # check if calibre converter-executable is existing
+            if not os.path.exists(config.config_converterpath):
+                self._handleError(N_("Calibre ebook-convert %(tool)s not found", tool=config.config_converterpath))
+                return
+            has_cover = local_db.get_book(book_id).has_cover
+            check, error_message = self._convert_calibre(file_path, format_old_ext, format_new_ext, has_cover)
+
+        if check == 0:
+            cur_book = local_db.get_book(book_id)
+            if os.path.isfile(file_path + format_new_ext):
+                new_format = local_db.session.query(db.Data).filter(db.Data.book == book_id) \
+                    .filter(db.Data.format == self.settings['new_book_format'].upper()).one_or_none()
+                if not new_format:
+                    new_format = db.Data(name=cur_book.data[0].name,
+                                         book_format=self.settings['new_book_format'].upper(),
+                                         book=book_id, uncompressed_size=os.path.getsize(file_path + format_new_ext))
+                    try:
+                        local_db.session.merge(new_format)
+                        local_db.session.commit()
+                        if self.settings['new_book_format'].upper() in ['KEPUB', 'EPUB', 'EPUB3']:
+                            ub_session = init_db_thread()
+                            remove_synced_book(book_id, True, ub_session)
+                            ub_session.close()
+                    except SQLAlchemyError as e:
+                        local_db.session.rollback()
+                        log.error("Database error: %s", e)
+                        local_db.session.close()
+                        self._handleError(error_message)
+                        return
+                self.results['path'] = cur_book.path
+                self.title = cur_book.title
+                self.results['title'] = self.title
+                if not config.config_use_google_drive:
+                    self._handleSuccess()
+                return os.path.basename(file_path + format_new_ext)
+            else:
+                error_message = N_('%(format)s format not found on disk', format=format_new_ext.upper())
+        local_db.session.close()
+        log.info("ebook converter failed with error while converting book")
+        if not error_message:
+            error_message = N_('Ebook converter failed with unknown error')
+        else:
+            log.error(error_message)
+        self._handleError(error_message)
+        return
+
+    def _convert_kepubify(self, file_path, format_old_ext, format_new_ext):
+        if config.config_embed_metadata and config.config_binariesdir:
+            tmp_dir, temp_file_name = helper.do_calibre_export(self.book_id, format_old_ext[1:])
+            filename = os.path.join(tmp_dir, temp_file_name + format_old_ext)
+            temp_file_path = tmp_dir
+        else:
+            filename = file_path + format_old_ext
+            temp_file_path = os.path.dirname(file_path)
+        quotes = [1, 3]
+        command = [config.config_kepubifypath, filename, '-o', temp_file_path, '-i']
+        try:
+            p = process_open(command, quotes)
+        except OSError as e:
+            return 1, N_("Kepubify-converter failed: %(error)s", error=e)
+        self.progress = 0.01
+        while True:
+            nextline = p.stdout.readlines()
+            nextline = [x.strip('\n') for x in nextline if x != '\n']
+            for line in nextline:
+                log.debug(line)
+            if p.poll() is not None:
+                break
+
+        # process returncode
+        check = p.returncode
+
+        # move file
+        if check == 0:
+            converted_file = glob.glob(glob.escape(os.path.splitext(filename)[0]) + "*.kepub.epub")
+            if len(converted_file) == 1:
+                copyfile(converted_file[0], (file_path + format_new_ext))
+                os.unlink(converted_file[0])
+            else:
+                return 1, N_("Converted file not found or more than one file in folder %(folder)s",
+                             folder=os.path.dirname(file_path))
+        return check, None
+
+    def _convert_calibre(self, file_path, format_old_ext, format_new_ext, has_cover):
+        path_tmp_opf = None
+        try:
+            # path_tmp_opf = self._embed_metadata()
+            if config.config_embed_metadata:
+                quotes = [5]
+                tmp_dir = get_temp_dir()
+                calibredb_binarypath = os.path.join(config.config_binariesdir, SUPPORTED_CALIBRE_BINARIES["calibredb"])
+                my_env = os.environ.copy()
+                if config.config_calibre_split:
+                    my_env['CALIBRE_OVERRIDE_DATABASE_PATH'] = os.path.join(config.config_calibre_dir, "metadata.db")
+                    library_path = config.config_calibre_split_dir
+                else:
+                    library_path = config.config_calibre_dir
+
+                opf_command = [calibredb_binarypath, 'show_metadata', '--as-opf', str(self.book_id),
+                               '--with-library', library_path]
+                p = process_open(opf_command, quotes, my_env, newlines=False)
+                lines = list()
+                while p.poll() is None:
+                    lines.append(p.stdout.readline())
+                check = p.returncode
+                calibre_traceback = p.stderr.readlines()
+                if check == 0:
+                    path_tmp_opf = os.path.join(tmp_dir, "metadata_" + str(uuid4()) + ".opf")
+                    with open(path_tmp_opf, 'wb') as fd:
+                        fd.write(b''.join(lines))
+                else:
+                    error_message = ""
+                    for ele in calibre_traceback:
+                        if not ele.startswith('Traceback') and not ele.startswith('  File'):
+                            error_message = N_("Calibre failed with error: %(error)s", error=ele)
+                    return check, error_message
+            quotes = [1, 2]
+            quotes_index = 3
+            command = [config.config_converterpath, (file_path + format_old_ext),
+                       (file_path + format_new_ext)]
+            if config.config_embed_metadata:
+                quotes.append(4)
+                quotes_index = 5
+                command.extend(['--from-opf', path_tmp_opf])
+                if has_cover:
+                    quotes.append(6)
+                    command.extend(['--cover', os.path.join(os.path.dirname(file_path), 'cover.jpg')])
+                    quotes_index = 7
+            if config.config_calibre:
+                parameters = re.findall(r"(--[\w-]+)(?:(\s(?:(\".+\")|(?:.+?)))(?:\s|$))?",
+                                        config.config_calibre, re.IGNORECASE | re.UNICODE)
+                if parameters:
+                    for param in parameters:
+                        command.append(strip_whitespaces(param[0]))
+                        quotes_index += 1
+                        if param[1] != "":
+                            parsed = strip_whitespaces(param[1]).strip("\"")
+                            command.append(parsed)
+                            quotes.append(quotes_index)
+                            quotes_index += 1
+            p = process_open(command, quotes, newlines=False)
+        except OSError as e:
+            return 1, N_("Ebook-converter failed: %(error)s", error=e)
+
+        while p.poll() is None:
+            nextline = p.stdout.readline()
+            if isinstance(nextline, bytes):
+                nextline = nextline.decode('utf-8', errors="ignore").strip('\r\n')
+            if nextline:
+                log.debug(nextline)
+            # parse progress string from calibre-converter
+            progress = re.search(r"(\d+)%\s.*", nextline)
+            if progress:
+                self.progress = int(progress.group(1)) / 100
+                if config.config_use_google_drive:
+                    self.progress *= 0.9
+
+        # process returncode
+        check = p.returncode
+        calibre_traceback = p.stderr.readlines()
+        error_message = ""
+        for ele in calibre_traceback:
+            ele = ele.decode('utf-8', errors="ignore").strip('\n')
+            log.debug(ele)
+            if not ele.startswith('Traceback') and not ele.startswith('  File'):
+                error_message = N_("Calibre failed with error: %(error)s", error=ele)
+        return check, error_message
+
+    @property
+    def name(self):
+        return N_("Convert")
+
+    def __str__(self):
+        if self.ereader_mail:
+            return "Convert Book {} and mail it to {}".format(self.book_id, self.ereader_mail)
+        else:
+            return "Convert Book {}".format(self.book_id)
+
+    @property
+    def is_cancellable(self):
+        return False

cps/tasks/mail.py

@@ -34,6 +34,7 @@ from cps.services import gmail
 from cps.embed_helper import do_calibre_export
 from cps import logger, config
 from cps import gdriveutils
+from cps.string_helper import strip_whitespaces
 import uuid
 
 log = logger.create()
@@ -127,9 +128,9 @@ class TaskEmail(CalibreTask):
         try:
             # Parse out the address from the From line, and then the domain from that
             from_email = parseaddr(self.settings["mail_from"])[1]
-            msgid_domain = from_email.partition('@')[2].strip()
+            msgid_domain = strip_whitespaces(from_email.partition('@')[2])
             # This can sometimes sneak through parseaddr if the input is malformed
-            msgid_domain = msgid_domain.rstrip('>').strip()
+            msgid_domain = strip_whitespaces(msgid_domain.rstrip('>'))
         except Exception:
             msgid_domain = ''
         return msgid_domain or 'calibre-web.com'

cps/tasks/thumbnail.py

@@ -20,14 +20,13 @@ import os
 from shutil import copyfile, copyfileobj
 from urllib.request import urlopen
 from io import BytesIO
+from datetime import datetime, timezone
 
 from .. import constants
 from cps import config, db, fs, gdriveutils, logger, ub
 from cps.services.worker import CalibreTask, STAT_CANCELLED, STAT_ENDED
-from datetime import datetime
 from sqlalchemy import func, text, or_
 from flask_babel import lazy_gettext as N_
-
 try:
     from wand.image import Image
     use_IM = True
@@ -36,7 +35,7 @@ except (ImportError, RuntimeError) as e:
 
 
 def get_resize_height(resolution):
-    return int(225 * resolution)
+    return int(255 * resolution)
 
 
 def get_resize_width(resolution, original_width, original_height):
@@ -73,7 +72,8 @@ class TaskGenerateCoverThumbnails(CalibreTask):
         self.cache = fs.FileSystem()
         self.resolutions = [
             constants.COVER_THUMBNAIL_SMALL,
-            constants.COVER_THUMBNAIL_MEDIUM
+            constants.COVER_THUMBNAIL_MEDIUM,
+            constants.COVER_THUMBNAIL_LARGE
         ]
 
     def run(self, worker_thread):
@@ -123,7 +123,7 @@ class TaskGenerateCoverThumbnails(CalibreTask):
             .query(ub.Thumbnail) \
             .filter(ub.Thumbnail.type == constants.THUMBNAIL_TYPE_COVER) \
             .filter(ub.Thumbnail.entity_id == book_id) \
-            .filter(or_(ub.Thumbnail.expiration.is_(None), ub.Thumbnail.expiration > datetime.utcnow())) \
+            .filter(or_(ub.Thumbnail.expiration.is_(None), ub.Thumbnail.expiration > datetime.now(timezone.utc))) \
             .all()
 
     def create_book_cover_thumbnails(self, book):
@@ -165,7 +165,7 @@ class TaskGenerateCoverThumbnails(CalibreTask):
             self.app_db_session.rollback()
 
     def update_book_cover_thumbnail(self, book, thumbnail):
-        thumbnail.generated_at = datetime.utcnow()
+        thumbnail.generated_at = datetime.now(timezone.utc)
 
         try:
             self.app_db_session.commit()
@@ -197,9 +197,11 @@ class TaskGenerateCoverThumbnails(CalibreTask):
                             img.format = thumbnail.format
                             img.save(filename=filename)
                         else:
-                            with open(filename, 'rb') as fd:
+                            stream.seek(0)
+                            with open(filename, 'wb') as fd:
                                 copyfileobj(stream, fd)
 
+
                 except Exception as ex:
                     # Bubble exception to calling function
                     self.log.debug('Error generating thumbnail file: ' + str(ex))
@@ -322,12 +324,12 @@ class TaskGenerateSeriesThumbnails(CalibreTask):
             .all()
 
     def get_series_thumbnails(self, series_id):
-        return self.app_db_session \
-            .query(ub.Thumbnail) \
-            .filter(ub.Thumbnail.type == constants.THUMBNAIL_TYPE_SERIES) \
-            .filter(ub.Thumbnail.entity_id == series_id) \
-            .filter(or_(ub.Thumbnail.expiration.is_(None), ub.Thumbnail.expiration > datetime.utcnow())) \
-            .all()
+        return (self.app_db_session
+            .query(ub.Thumbnail)
+            .filter(ub.Thumbnail.type == constants.THUMBNAIL_TYPE_SERIES)
+            .filter(ub.Thumbnail.entity_id == series_id)
+            .filter(or_(ub.Thumbnail.expiration.is_(None), ub.Thumbnail.expiration > datetime.now(timezone.utc)))
+            .all())
 
     def create_series_thumbnail(self, series, series_books, resolution):
         thumbnail = ub.Thumbnail()
@@ -346,7 +348,7 @@ class TaskGenerateSeriesThumbnails(CalibreTask):
             self.app_db_session.rollback()
 
     def update_series_thumbnail(self, series_books, thumbnail):
-        thumbnail.generated_at = datetime.utcnow()
+        thumbnail.generated_at = datetime.now(timezone.utc)
 
         try:
             self.app_db_session.commit()

cps/templates/author.html

@@ -62,18 +62,16 @@
               <a class="author-name" href="{{url_for('web.books_list',  data='author', sort_param='stored', book_id=author.id) }}">{{author.name.replace('|',',')|shortentitle(30)}}</a>
             {% endif %}
           {% endfor %}
-          {% for format in entry.Books.data %}
-            {% if format.format|lower in g.constants.EXTENSIONS_AUDIO %}
+          {% if entry.Books.data|music %}
             <span class="glyphicon glyphicon-music"></span>
-            {% endif %}
-          {% endfor %}
+          {% endif %}
         </p>
         {% if entry.Books.series.__len__() > 0 %}
         <p class="series">
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.Books.series[0].id )}}">
             {{entry.Books.series[0].name}}
           </a>
-          ({{entry.Books.series_index|formatseriesindex}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
         {% if entry.Books.ratings.__len__() > 0 %}
@@ -124,7 +122,7 @@
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.series[0].id )}}">
             {{entry.series[0].name}}
           </a>
-          ({{entry.series_index|formatseriesindex}})
+          ({{entry.series_index|formatfloat(2)}})
         </p>
         {% endif %}
         <div class="rating">

cps/templates/book_edit.html

@@ -47,26 +47,41 @@
       </form>
   </div>
 {% endif %}
+{% if current_user.role_upload() and g.allow_upload %}
 
+  <div class="text-center more-stuff"><!--h4 aria-label="Upload new book format"></h4-->
+  <form id="form-upload-format" action="{{ url_for('edit-book.upload') }}" data-title="{{_('Uploading...')}}" data-footer="{{_('Close')}}" data-failed="{{_('Error')}}" data-message="{{_('Upload done, processing, please wait...')}}" method="post" enctype="multipart/form-data">
+    <div class="text-center">
+      <input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
+      <input type="hidden" name="book_id" value="{{ book.id }}">
+      <div role="group" aria-label="Upload new book format">
+        <label class="btn btn-primary btn-file" for="btn-upload-format">{{ _('Upload Format') }}</label>
+        <div class="upload-format-input-text" id="upload-format"></div>
+        <input id="btn-upload-format" name="btn-upload-format" type="file" accept="{% for format in accept %}.{% if format != ''%}{{format}}{% else %}*{% endif %}{{ ',' if not loop.last }}{% endfor %}" multiple>
+      </div>
+    </div>
+  </form>
   </div>
+{% endif %}
+</div>
 <form role="form" action="{{ url_for('edit-book.edit_book', book_id=book.id) }}" method="post" enctype="multipart/form-data" id="book_edit_frm">
   <input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
   <div class="col-sm-9 col-xs-12">
     <div class="form-group">
-      <label for="book_title">{{_('Book Title')}}</label>
-      <input type="text" class="form-control" name="book_title" id="book_title" value="{{book.title}}">
+      <label for="title">{{_('Book Title')}}</label>
+      <input type="text" class="form-control" name="title" id="title" value="{{book.title}}">
     </div>
     <div class="text-center">
         <button type="button" class="btn btn-default" id="xchange" ><span class="glyphicon glyphicon-arrow-up"></span><span class="glyphicon glyphicon-arrow-down"></span></button>
     </div>
     <div id="author_div" class="form-group">
       <label for="bookAuthor">{{_('Author')}}</label>
-      <input type="text" class="form-control typeahead" autocomplete="off" name="author_name" id="bookAuthor" value="{{' & '.join(authors)}}">
+      <input type="text" class="form-control typeahead" autocomplete="off" name="authors" id="authors" value="{{' & '.join(authors)}}">
     </div>
 
     <div class="form-group">
-      <label for="description">{{_('Description')}}</label>
-      <textarea class="form-control" name="description" id="description" rows="7">{% if book.comments %}{{book.comments[0].text}}{%endif%}</textarea>
+      <label for="comments">{{_('Description')}}</label>
+      <textarea class="form-control" name="comments" id="comments" rows="7">{% if book.comments %}{{book.comments[0].text}}{%endif%}</textarea>
     </div>
 
     <div class="form-group">
@@ -93,7 +108,7 @@
     </div>
     <div class="form-group">
       <label for="series_index">{{_('Series ID')}}</label>
-      <input type="number" step="0.01" min="0" placeholder="1" class="form-control" name="series_index" id="series_index" value="{{book.series_index}}">
+      <input type="number" step="0.01" min="0" placeholder="1" class="form-control" name="series_index" id="series_index" value="{{book.series_index|formatfloat(2)}}">
     </div>
     <div class="form-group">
       <label for="rating">{{_('Rating')}}</label>
@@ -196,13 +211,6 @@
           </div>
         {% endfor %}
       {% endif %}
-      {% if current_user.role_upload() and g.allow_upload %}
-          <div role="group" aria-label="Upload new book format">
-            <label class="btn btn-primary btn-file" for="btn-upload-format">{{ _('Upload Format') }}</label>
-            <div class="upload-format-input-text" id="upload-format"></div>
-            <input id="btn-upload-format" name="btn-upload-format" type="file">
-          </div>
-      {% endif %}
 
     <div class="checkbox">
       <label>
@@ -288,7 +296,7 @@
     'no_result': {{_('No Result(s) found! Please try another keyword.')|safe|tojson}},
     'author': {{_('Author')|safe|tojson}},
     'publisher': {{_('Publisher')|safe|tojson}},
-    'description': {{_('Description')|safe|tojson}},
+    'comments': {{_('Description')|safe|tojson}},
     'source': {{_('Source')|safe|tojson}},
   };
   var language = '{{ current_user.locale }}';

cps/templates/book_table.html

@@ -66,7 +66,7 @@
             {{ text_table_row('authors', _('Enter Authors'),_('Authors'), true, true) }}
             {{ text_table_row('tags', _('Enter Categories'),_('Categories'), false, true) }}
             {{ text_table_row('series', _('Enter Series'),_('Series'), false, true) }}
-            <th data-field="series_index" id="series_index" data-visible="{{visiblility.get('series_index')}}" data-edit-validate="{{ _('This Field is Required') }}" data-sortable="true" {% if current_user.role_edit() %} data-editable-type="number" data-editable-placeholder="1" data-editable-step="0.01" data-editable-min="0" data-editable-url="{{ url_for('edit-book.edit_list_book', param='series_index')}}" data-edit="true" data-editable-title="{{_('Enter Title')}}"{% endif %}>{{_('Series Index')}}</th>
+            <th data-field="series_index" id="series_index" data-visible="{{visiblility.get('series_index')}}"  data-formatter="seriesIndexFormatter" data-edit-validate="{{ _('This Field is Required') }}" data-sortable="true" {% if current_user.role_edit() %} data-editable-type="number" data-editable-placeholder="1" data-editable-step="0.01" data-editable-min="0" data-editable-url="{{ url_for('edit-book.edit_list_book', param='series_index')}}" data-edit="true" data-editable-title="{{_('Enter Title')}}"{% endif %}>{{_('Series Index')}}</th>
             {{ text_table_row('languages', _('Enter Languages'),_('Languages'), false, true) }}
             <!--th data-field="pubdate" data-type="date" data-visible="{{visiblility.get('pubdate')}}" data-viewformat="dd.mm.yyyy" id="pubdate" data-sortable="true">{{_('Publishing Date')}}</th-->
             {{ text_table_row('publishers', _('Enter Publishers'),_('Publishers'), false, true) }}

cps/templates/detail.html

@@ -20,7 +20,7 @@
                                         {{ _('Download') }} :
                                     </button>
                                     {% for format in entry.data %}
-                                        <a href="{{ url_for('web.download_link', book_id=entry.id, book_format=format.format|lower, anyname=entry.id|string+'.'+format.format|lower) }}"
+                                        <a href="{{ url_for('web.download_link', book_id=entry.id, book_format=format.format|lower, anyname=entry.id|string+'.'+format.format|lower|replace('kepub', 'kepub.epub')) }}"
                                            id="btnGroupDrop1{{ format.format|lower }}" class="btn btn-primary"
                                            role="button">
                                             <span class="glyphicon glyphicon-download"></span>{{ format.format }}
@@ -36,7 +36,7 @@
                                     <ul class="dropdown-menu" aria-labelledby="btnGroupDrop1">
                                         {% for format in entry.data %}
                                             <li>
-                                                <a href="{{ url_for('web.download_link', book_id=entry.id, book_format=format.format|lower, anyname=entry.id|string+'.'+format.format|lower) }}">{{ format.format }}
+                                                <a href="{{ url_for('web.download_link', book_id=entry.id, book_format=format.format|lower, anyname=entry.id|string+'.'+format.format|lower|replace('kepub', 'kepub.epub')) }}">{{ format.format }}
                                                     ({{ format.uncompressed_size|filesizeformat }})</a></li>
                                         {% endfor %}
                                     </ul>
@@ -147,7 +147,7 @@
                 </div>
             {% endif %}
             {% if entry.series|length > 0 %}
-                <p>{{ _("Book %(index)s of %(range)s", index=entry.series_index | formatfloat(2), range=(url_for('web.books_list', data='series', sort_param='stored', book_id=entry.series[0].id)|escapedlink(entry.series[0].name))|safe) }}</p>
+                <p>{{ _("Book %(index)s of %(range)s", index=entry.series_index|formatfloat(2), range=(url_for('web.books_list', data='series', sort_param='stored', book_id=entry.series[0].id)|escapedlink(entry.series[0].name))|safe) }}</p>
 
             {% endif %}
 

cps/templates/image.html

@@ -15,6 +15,6 @@
     <img
         srcset="{{ srcset }}"
         src="{{ url_for('web.get_series_cover', series_id=series.id, resolution='og', c='day'|cache_timestamp) }}"
-        alt="{{ book_title }}"
+        alt="{{ title }}"
     />
 {%- endmacro %}

cps/templates/index.html

@@ -42,7 +42,7 @@
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.Books.series[0].id )}}">
             {{entry.Books.series[0].name}}
           </a>
-          ({{entry.Books.series_index|formatseriesindex}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
         {% if entry.Books.ratings.__len__() > 0 %}
@@ -119,11 +119,9 @@
               <a class="author-name" href="{{url_for('web.books_list', data='author', book_id=author.id, sort_param='stored') }}">{{author.name.replace('|',',')|shortentitle(30)}}</a>
             {% endif %}
           {% endfor %}
-          {% for format in entry.Books.data %}
-            {% if format.format|lower in g.constants.EXTENSIONS_AUDIO %}
+          {% if entry.Books.data|music %}
             <span class="glyphicon glyphicon-music"></span>
-            {% endif %}
-          {%endfor%}
+          {% endif %}
         </p>
         {% if entry.Books.series.__len__() > 0 %}
         <p class="series">
@@ -134,7 +132,7 @@
         {% else %}
             <span>{{entry.Books.series[0].name}}</span>
         {% endif %}
-          ({{entry.Books.series_index|formatseriesindex}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
         {% if entry.Books.ratings.__len__() > 0 %}

cps/templates/layout.html

@@ -80,6 +80,7 @@
                       <div class="form-group">
                         <span class="btn btn-default btn-file">{{_('Upload')}}<input id="btn-upload" name="btn-upload"
                         type="file" accept="{% for format in accept %}.{% if format != ''%}{{format}}{% else %}*{% endif %}{{ ',' if not loop.last }}{% endfor %}" multiple></span>
+                        <input class="hide" id="btn-upload2" name="btn-upload2" type="file" accept="{% for format in accept %}.{% if format != ''%}{{format}}{% else %}*{% endif %}{{ ',' if not loop.last }}{% endfor %}">
                       </div>
                     </form>
                   </li>

cps/templates/listenmp3.html

@@ -59,7 +59,7 @@
         </div>
       {% endif %}
       {% if entry.series|length > 0 %}
-          <p>{{_("Book %(index)s of %(range)s", index=entry.series_index | formatfloat(2), range=(url_for('web.books_list', data='series', sort_param='stored', book_id=entry.series[0].id)|escapedlink(entry.series[0].name))|safe)}}</p>
+          <p>{{_("Book %(index)s of %(range)s", index=entry.series_index|formatfloat(2), range=(url_for('web.books_list', data='series', sort_param='stored', book_id=entry.series[0].id)|escapedlink(entry.series[0].name))|safe)}}</p>
 
       {% endif %}
 

cps/templates/locales/en/translation.json

@@ -1,10 +0,0 @@
-{
-  "input": {
-    "placeholder": "a placeholder"
-  },
-  "nav": {
-    "home": "Home",
-    "page1": "Page One",
-    "page2": "Page Two"
-  }
-}

cps/templates/read.html

@@ -135,19 +135,23 @@
         window.themes = {
             "darkTheme": {
                 "bgColor": "#202124",
-                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}"
+                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}",
+                "title-color": "#fff"
             },
             "lightTheme": {
                 "bgColor": "white",
-                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}"
+                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}",
+                "title-color": "#4f4f4f"
             },
             "sepiaTheme": {
                 "bgColor": "#ece1ca",
-                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}"
+                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}",
+                "title-color": "#4f4f4f"
             },
             "blackTheme": {
                 "bgColor": "black",
-                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}"
+                "css_path": "{{ url_for('static', filename='css/epub_themes.css') }}",
+                "title-color": "#fff"
             },
         };
 
@@ -170,6 +174,7 @@
 
             // Apply theme to rest of the page. 
             document.getElementById("main").style.backgroundColor = themes[id]["bgColor"];
+            document.getElementById("titlebar").style.color = themes[id]["title-color"] || "#fff";
         }
 
         // font size settings logic

cps/templates/search.html

@@ -73,18 +73,16 @@
               <a class="author-name" href="{{url_for('web.books_list',  data='author', sort_param='stored', book_id=author.id) }}">{{author.name.replace('|',',')|shortentitle(30)}}</a>
             {% endif %}
           {% endfor %}
-          {% for format in entry.Books.data %}
-            {% if format.format|lower in g.constants.EXTENSIONS_AUDIO %}
+          {% if entry.Books.data|music %}
             <span class="glyphicon glyphicon-music"></span>
-            {% endif %}
-          {% endfor %}
+          {% endif %}
         </p>
         {% if entry.Books.series.__len__() > 0 %}
         <p class="series">
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.Books.series[0].id )}}">
             {{entry.Books.series[0].name}}
           </a>
-          ({{entry.Books.series_index|formatseriesindex}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
 

cps/templates/search_form.html

@@ -5,12 +5,12 @@
   <form role="form" id="search" action="{{ url_for('search.advanced_search_form') }}" method="POST">
     <input type="hidden" name="csrf_token" value="{{ csrf_token() }}">
     <div class="form-group">
-      <label for="book_title">{{_('Book Title')}}</label>
-      <input type="text" class="form-control" name="book_title" id="book_title" value="">
+      <label for="title">{{_('Book Title')}}</label>
+      <input type="text" class="form-control" name="title" id="title" value="">
     </div>
     <div class="form-group">
-      <label for="bookAuthor">{{_('Author')}}</label>
-      <input type="text" class="form-control typeahead" name="author_name" id="bookAuthor" value="" autocomplete="off">
+      <label for="authors">{{_('Author')}}</label>
+      <input type="text" class="form-control typeahead" name="authors" id="authors" value="" autocomplete="off">
     </div>
     <div class="form-group">
       <label for="Publisher">{{_('Publisher')}}</label>
@@ -151,8 +151,8 @@
         </div>
     </div>
     <div class="form-group">
-      <label for="comment">{{_('Description')}}</label>
-      <input type="text" class="form-control" name="comment" id="comment" value="">
+      <label for="comments">{{_('Description')}}</label>
+      <input type="text" class="form-control" name="comments" id="comments" value="">
     </div>
 
     {% if cc|length > 0 %}

cps/templates/shelf.html

@@ -15,16 +15,19 @@
       </form>
       {% if entries.__len__() %}
       <a id="order_shelf" href="{{ url_for('shelf.order_shelf', shelf_id=shelf.id) }}" class="btn btn-primary">{{ _('Arrange books manually') }} </a>
-      <button id="toggle_order_shelf" type="button" data-alt-text="{{ _('Disable Change order') }}" class="btn btn-primary">{{ _('Enable Change order') }}</button>
+      <button id="toggle_order_shelf" data-alt-text="{% if status == "on" %}{{_('Disable Change order')}}{% else %}{{_('Enable Change order')}}{% endif %}" data-view="shelf" class="btn btn-primary{% if status == 'on' %} dummy">{{ _('Enable Change order') }}{% else%}">{{ _('Disable Change order') }}{% endif %}</button>
         <div class="filterheader hidden-xs">
-          <a data-toggle="tooltip" title="{{_('Sort according to book date, newest first')}}" id="new" class="btn btn-primary disabled" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='new')}}"><span class="glyphicon glyphicon-book"></span> <span class="glyphicon glyphicon-calendar"></span><span class="glyphicon glyphicon-sort-by-order"></span></a>
-          <a data-toggle="tooltip" title="{{_('Sort according to book date, oldest first')}}" id="old" class="btn btn-primary disabled" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='old')}}"><span class="glyphicon glyphicon-book"></span> <span class="glyphicon glyphicon-calendar"></span><span class="glyphicon glyphicon-sort-by-order-alt"></span></a>
-          <a data-toggle="tooltip" title="{{_('Sort title in alphabetical order')}}" id="asc" class="btn btn-primary disabled" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='abc')}}"><span class="glyphicon glyphicon-font"></span><span class="glyphicon glyphicon-sort-by-alphabet"></span></a>
-          <a data-toggle="tooltip" title="{{_('Sort title in reverse alphabetical order')}}" id="desc" class="btn btn-primary disabled" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='zyx')}}"><span class="glyphicon glyphicon-font"></span><span class="glyphicon glyphicon-sort-by-alphabet-alt"></span></a>
-          <a data-toggle="tooltip" title="{{_('Sort authors in alphabetical order')}}" id="auth_az" class="btn btn-primary disabled" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='authaz')}}"><span class="glyphicon glyphicon-user"></span><span class="glyphicon glyphicon-sort-by-alphabet"></span></a>
-          <a data-toggle="tooltip" title="{{_('Sort authors in reverse alphabetical order')}}" id="auth_za" class="btn btn-primary disabled" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='authza')}}"><span class="glyphicon glyphicon-user"></span><span class="glyphicon glyphicon-sort-by-alphabet-alt"></span></a>
-          <a data-toggle="tooltip" title="{{_('Sort according to publishing date, newest first')}}" id="pub_new" class="btn btn-primary disabled" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='pubnew')}}"><span class="glyphicon glyphicon-calendar"></span><span class="glyphicon glyphicon-sort-by-order"></span></a>
-          <a data-toggle="tooltip" title="{{_('Sort according to publishing date, oldest first')}}" id="pub_old" class="btn btn-primary disabled" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='pubold')}}"><span class="glyphicon glyphicon-calendar"></span><span class="glyphicon glyphicon-sort-by-order-alt"></span></a>
+          <a data-toggle="tooltip" title="{{_('Sort according to book date, newest first')}}" id="new" class="btn btn-primary {% if order == "new" %} active{% endif %}{% if status == "on" %} disabled{% endif %}" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='new')}}"><span class="glyphicon glyphicon-book"></span> <span class="glyphicon glyphicon-calendar"></span><span class="glyphicon glyphicon-sort-by-order"></span></a>
+          <a data-toggle="tooltip" title="{{_('Sort according to book date, oldest first')}}" id="old" class="btn btn-primary {% if order == 'old' %} active{% endif %}{% if status == 'on' %} disabled{% endif %}" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='old')}}"><span class="glyphicon glyphicon-book"></span> <span class="glyphicon glyphicon-calendar"></span><span class="glyphicon glyphicon-sort-by-order-alt"></span></a>
+          <a data-toggle="tooltip" title="{{_('Sort title in alphabetical order')}}" id="asc" class="btn btn-primary {% if order == 'abc' %} active{% endif %}{% if status == 'on' %} disabled{% endif %}" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='abc')}}"><span class="glyphicon glyphicon-font"></span><span class="glyphicon glyphicon-sort-by-alphabet"></span></a>
+          <a data-toggle="tooltip" title="{{_('Sort title in reverse alphabetical order')}}" id="desc" class="btn btn-primary {% if order == 'zyx' %} active{% endif %}{% if status == 'on' %} disabled{% endif %}" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='zyx')}}"><span class="glyphicon glyphicon-font"></span><span class="glyphicon glyphicon-sort-by-alphabet-alt"></span></a>
+          <a data-toggle="tooltip" title="{{_('Sort authors in alphabetical order')}}" id="auth_az" class="btn btn-primary {% if order == 'authaz' %} active{% endif %}{% if status == 'on' %} disabled{% endif %}" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='authaz')}}"><span class="glyphicon glyphicon-user"></span><span class="glyphicon glyphicon-sort-by-alphabet"></span></a>
+          <a data-toggle="tooltip" title="{{_('Sort authors in reverse alphabetical order')}}" id="auth_za" class="btn btn-primary {% if order == 'authza' %} active{% endif %}{% if status == 'on' %} disabled{% endif %}" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='authza')}}"><span class="glyphicon glyphicon-user"></span><span class="glyphicon glyphicon-sort-by-alphabet-alt"></span></a>
+          <a data-toggle="tooltip" title="{{_('Sort according to publishing date, newest first')}}" id="pub_new" class="btn btn-primary {% if order == 'pubnew' %} active{% endif %}{% if status == 'on' %} disabled{% endif %}" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='pubnew')}}"><span class="glyphicon glyphicon-calendar"></span><span class="glyphicon glyphicon-sort-by-order"></span></a>
+          <a data-toggle="tooltip" title="{{_('Sort according to publishing date, oldest first')}}" id="pub_old" class="btn btn-primary {% if order == 'pubold' %} active{% endif %}{% if status == 'on' %} disabled{% endif %}" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='pubold')}}"><span class="glyphicon glyphicon-calendar"></span><span class="glyphicon glyphicon-sort-by-order-alt"></span></a>
+          <a data-toggle="tooltip" title="{{_('Sort according to book added to shelf, newest first')}}" id="shelf_new" class="btn btn-primary {% if order == 'shelfnew' %} active{% endif %}{% if status == 'on' %} disabled{% endif %}" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='shelfnew')}}"><span class="glyphicon glyphicon-list"></span><span class="glyphicon glyphicon-sort-by-order"></span></a>
+          <a data-toggle="tooltip" title="{{_('Sort according to book added to shelf, oldest first')}}" id="shelf_old" class="btn btn-primary {% if order == 'shelfold' %} active{% endif %}{% if status == 'on' %} disabled{% endif %}" href="{{url_for('shelf.show_shelf', shelf_id=shelf.id, sort_param='shelfold')}}"><span class="glyphicon glyphicon-list"></span><span class="glyphicon glyphicon-sort-by-order-alt"></span></a
+          >
         </div>
       {% endif %}
     {% endif %}
@@ -67,7 +70,7 @@
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.Books.series[0].id )}}">
             {{entry.Books.series[0].name}}
           </a>
-          ({{entry.Books.series_index|formatseriesindex}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
         {% if entry.Books.ratings.__len__() > 0 %}

cps/templates/shelf_order.html

@@ -20,7 +20,7 @@
                   {{entry['Books']['title']}}
                   {% if entry['Books']['series']|length > 0 %}
                     <br>
-                    {{entry['Books']['series_index']}} - {{entry['Books']['series'][0].name}}
+                    {{entry['Books']['series_index']|formatfloat(2)}} - {{entry['Books']['series'][0].name}}
                   {% endif %}
                   <br>
                   {% for author in entry['Books']['author'] %}

cps/templates/shelfdown.html

@@ -45,7 +45,7 @@
           <a href="{{url_for('web.books_list', data='series', sort_param='stored', book_id=entry.Books.series[0].id )}}">
             {{entry.Books.series[0].name}}
           </a>
-          ({{entry.Books.series_index}})
+          ({{entry.Books.series_index|formatfloat(2)}})
         </p>
         {% endif %}
       </div>
@@ -55,7 +55,7 @@
             {% if entry.Books.data|length %}
             <div class="btn-group" role="group">
                 {% for format in entry.Books.data %}
-                <a href="{{ url_for('web.download_link', book_id=entry.Books.id, book_format=format.format|lower, anyname=entry.Books.id|string+'.'+format.format|lower) }}" id="btnGroupDrop{{entry.Books.id}}{{format.format|lower}}" class="btn btn-primary" role="button">
+                <a href="{{ url_for('web.download_link', book_id=entry.Books.id, book_format=format.format|lower, anyname=entry.Books.id|string+'.'+format.format|lower|replace('kepub', 'kepub.epub')) }}" id="btnGroupDrop{{entry.Books.id}}{{format.format|lower}}" class="btn btn-primary" role="button">
                   <span class="glyphicon glyphicon-download"></span>{{format.format}} ({{ format.uncompressed_size|filesizeformat }})
                 </a>
                 {% endfor %}

cps/templates/user_edit.html

@@ -25,7 +25,7 @@
         </div>
     {% endif %}
     <div class="form-group">
-      <label for="kindle_mail">{{_('Send to eReader Email Address. Use comma to seperate emails for multiple eReaders')}}</label>
+      <label for="kindle_mail">{{_('Send to eReader Email Address. Use comma to separate emails for multiple eReaders')}}</label>
       <input type="email" class="form-control" name="kindle_mail" id="kindle_mail" value="{{ content.kindle_mail if content.kindle_mail != None }}">
     </div>
     {% if not content.role_anonymous() %}