Updated testresults

Version bump for release
Merge remote-tracking branch 'it/patch-15' into master
2020-09-27 07:15:03 +02:00 · 2020-09-26 16:19:34 +02:00 · 2020-09-26 16:13:53 +02:00 · 2020-09-26 07:51:29 +02:00 · 2020-09-24 11:02:12 +02:00 · 2020-09-24 10:54:06 +02:00
163 changed files with 27176 additions and 13436 deletions
@@ -6,6 +6,7 @@ labels: ''
 assignees: ''

 ---
+<!-- Please have a look at our [Contributing Guidelines](https://github.com/janeczku/calibre-web/blob/master/CONTRIBUTING.md) -->

 **Describe the bug/problem**
 A clear and concise description of what the bug is. If you are asking for support, please check our [Wiki](https://github.com/janeczku/calibre-web/wiki) if your question is already answered there.
@@ -27,12 +28,12 @@ A clear and concise description of what you expected to happen.
 If applicable, add screenshots to help explain your problem.

 **Environment (please complete the following information):**
- - OS: [e.g. Windows 10/raspian]
- - Python version [e.g. python2.7]
- - Calibre-Web version [e.g. 0.6.5 or master@16.02.20, 19:55 ]:
- - Docker container [ None/Technosoft2000/Linuxuser]:
- - Special Hardware [e.g. Rasperry Pi Zero]
- - Browser [e.g. chrome, safari]
+ - OS: [e.g. Windows 10/Raspberry Pi OS]
+ - Python version: [e.g. python2.7]
+ - Calibre-Web version: [e.g. 0.6.8 or 087c4c59 (git rev-parse --short HEAD)]:
+ - Docker container: [None/Technosoft2000/Linuxuser]:
+ - Special Hardware: [e.g. Rasperry Pi Zero]
+ - Browser: [e.g. Chrome 83.0.4103.97, Safari 13.3.7, Firefox 68.0.1 ESR]

 **Additional context**
-Add any other context about the problem here. [e.g. access via reverse proxy]
+Add any other context about the problem here. [e.g. access via reverse proxy, database background sync, special database location]
@@ -7,6 +7,8 @@ assignees: ''

 ---

+<!-- Please have a look at our [Contributing Guidelines](https://github.com/janeczku/calibre-web/blob/master/CONTRIBUTING.md) -->
+
 **Is your feature request related to a problem? Please describe.**
 A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

@@ -1,49 +1,46 @@
 ## How to contribute to Calibre-Web

-First of all, we would like to thank you for reading this text. we are happy you are willing to contribute to Calibre-Web
+First of all, we would like to thank you for reading this text. We are happy you are willing to contribute to Calibre-Web.

 ### **General**

-**Communication language** is english. Google translated texts are not as bad as you might think, they are usually understandable, so don't worry if you generate your post that way.
+**Communication language** is English. Google translated texts are not as bad as you might think, they are usually understandable, so don't worry if you generate your post that way.

 **Calibre-Web** is not **Calibre**. If you are having a question regarding Calibre please post this at their [repository](https://github.com/kovidgoyal/calibre).

-**Docker-Containers** of Calibre-Web are maintained by different persons than the people who drive Calibre-Web. If we come to the conclusion during our analysis that the problem is related to Docker, we might ask you to open a new issue at the reprository of the Docker Container.
+**Docker-Containers** of Calibre-Web are maintained by different persons than the people who drive Calibre-Web. If we come to the conclusion during our analysis that the problem is related to Docker, we might ask you to open a new issue at the repository of the Docker Container.

-If you are having **Basic Installation Problems** with python or it's dependencies, please consider using your favorite search engine to find a solution. In case you can't find a solution, we are happy to help you.
+If you are having **Basic Installation Problems** with python or its dependencies, please consider using your favorite search engine to find a solution. In case you can't find a solution, we are happy to help you.

-We can offer only very limited support regarding configuration of **Reverse-Proxy Installations**, **OPDS-Reader** or other programs in combination with Calibre-Web. 
+We can offer only very limited support regarding configuration of **Reverse-Proxy Installations**, **OPDS-Reader** or other programs in combination with Calibre-Web.

 ### **Translation**

-Some of the user languages in Calibre-Web having missing translations. We are happy to add the missing texts if you are translate them. Create a Pull Request, create an issue with the .po file attached, or write an email to "ozzie.fernandez.isaacs@googlemail.com" with attached translation file. To display all book languages in your native language an additional file is used (iso_language_names.py). The content of this file is autogenerated with the corresponding translations of Calibre, please do not edit this file on your own. 
+Some of the user languages in Calibre-Web having missing translations. We are happy to add the missing texts if you translate them. Create a Pull Request, create an issue with the .po file attached, or write an email to "ozzie.fernandez.isaacs@googlemail.com" with attached translation file. To display all book languages in your native language an additional file is used (iso_language_names.py). The content of this file is auto-generated with the corresponding translations of Calibre, please do not edit this file on your own.

 ### **Documentation**

-The Calibre-Web documentation is hosted in the Github [Wiki](https://github.com/janeczku/calibre-web/wiki). The Wiki is open to everybody, if you find a problem, feel free to correct it. If information is missing, you are welcome to add it. The content will be reviewed time by time. Please try to be consitent with the form with the other Wiki pages (e.g. the project name is Calibre-Web with 2 capital letters and a dash in between).
+The Calibre-Web documentation is hosted in the Github [Wiki](https://github.com/janeczku/calibre-web/wiki). The Wiki is open to everybody, if you find a problem, feel free to correct it. If information is missing, you are welcome to add it. The content will be reviewed time by time. Please try to be consistent with the form with the other Wiki pages (e.g. the project name is Calibre-Web with 2 capital letters and a dash in between).

 ### **Reporting a bug**

-Do not open up a GitHub issue if the bug is a **security vulnerability** in Calibre-Web. Please write intead an email to "ozzie.fernandez.isaacs@googlemail.com".
+Do not open up a GitHub issue if the bug is a **security vulnerability** in Calibre-Web. Instead, please write an email to "ozzie.fernandez.isaacs@googlemail.com".

 Ensure the ***bug was not already reported** by searching on GitHub under [Issues](https://github.com/janeczku/calibre-web/issues). Please also check if a solution for your problem can be found in the [wiki](https://github.com/janeczku/calibre-web/wiki).

-If you're unable to find an **open issue** addressing the problem, open a [new one](https://github.com/janeczku/calibre-web/issues/new?assignees=&labels=&template=bug_report.md&title=). Be sure to include a **title** and **clear description**, as much relevant information as possible, the **issue form** helps you providing the right information. Deleting the form and just pasting the stack trace doesn't speed up fixing the problem. If your issue could be resolved, consider closing the issue. 
+If you're unable to find an **open issue** addressing the problem, open a [new one](https://github.com/janeczku/calibre-web/issues/new?assignees=&labels=&template=bug_report.md&title=). Be sure to include a **title** and **clear description**, as much relevant information as possible, the **issue form** helps you providing the right information. Deleting the form and just pasting the stack trace doesn't speed up fixing the problem. If your issue could be resolved, consider closing the issue.

 ### **Feature Request**

-If there is afeature missing in Calibre-Web and you can't find a feature request in the [Issues](https://github.com/janeczku/calibre-web/issues) section, you could create a [feature request](https://github.com/janeczku/calibre-web/issues/new?assignees=&labels=&template=feature_request.md&title=). 
-We will not extend Calibre-Web with any more login abilitys or add further files storages, or file syncing ability. Furthermore Calibre-Web is made for home usage for company inhouse usage, so requests regarding any sorts of social interaction capability, payment routines, search engine or web site analytics integration will not be implemeted. 
+If there is a feature missing in Calibre-Web and you can't find a feature request in the [Issues](https://github.com/janeczku/calibre-web/issues) section, you could create a [feature request](https://github.com/janeczku/calibre-web/issues/new?assignees=&labels=&template=feature_request.md&title=).
+We will not extend Calibre-Web with any more login abilities or add further files storages, or file syncing ability. Furthermore Calibre-Web is made for home usage for company in-house usage, so requests regarding any sorts of social interaction capability, payment routines, search engine or web site analytics integration will not be implemented.

 ### **Contributing code to Calibre-Web**

 Open a new GitHub pull request with the patch. Ensure the PR description clearly describes the problem and solution. Include the relevant issue number if applicable.

-In case your code enhances features of Calibre-Web: Create your pull request for the development branch if your enhancement consits of more than some lines of code in a local section of Calibre-Webs code. This makes it easier to test it and check all implication before it's made public. 
-
-Please check if your code runs on Python 2.7 (still necessary in 2020) and mainly on python 3. If possible and the feature is related to operating system functions, try to check it on Windows and Linux. 
-Calibre-Web is automatically tested on Linux in combination with python 3.7. The code for testing is in a [seperate repo](https://github.com/OzzieIsaacs/calibre-web-test) on Github. It uses unittest and performs real system tests with selenium, would be great if you could consider also writing some tests. 
-A static code analysis is done by Codacy, but it's partly broken and doesn't run automatically. You could check your code with ESLint before contributing, a configuration file can be found in the projects root folder. 
-
-
+In case your code enhances features of Calibre-Web: Create your pull request for the development branch if your enhancement consists of more than some lines of code in a local section of Calibre-Webs code. This makes it easier to test it and check all implication before it's made public.

+Please check if your code runs on Python 2.7 (still necessary in 2020) and mainly on python 3. If possible and the feature is related to operating system functions, try to check it on Windows and Linux.
+Calibre-Web is automatically tested on Linux in combination with python 3.7. The code for testing is in a [separate repo](https://github.com/OzzieIsaacs/calibre-web-test) on Github. It uses unit tests and performs real system tests with selenium; it would be great if you could consider also writing some tests.
+A static code analysis is done by Codacy, but it's partly broken and doesn't run automatically. You could check your code with ESLint before contributing, a configuration file can be found in the projects root folder.
@@ -54,23 +54,21 @@ Optionally, to enable on-the-fly conversion from one ebook format to another whe

 [Download and install](https://calibre-ebook.com/download) the Calibre desktop program for your platform and enter the folder including program name (normally /opt/calibre/ebook-convert, or C:\Program Files\calibre\ebook-convert.exe) in the field "calibre's converter tool" on the setup page.

-\*** DEPRECATED \*** Support will be removed in future releases
-
-[Download](http://www.amazon.com/gp/feature.html?docId=1000765211) Amazon's KindleGen tool for your platform and place the binary named `kindlegen` in the `vendor` folder.
+[Download](https://github.com/geek1011/kepubify/releases/tag/v3.1.2) Kepubify tool for your platform and place the binary starting with `kepubify` in Linux: `\opt\kepubify` Windows: `C:\Program Files\kepubify`.

 ## Docker Images

 Pre-built Docker images are available in these Docker Hub repositories:

 #### **Technosoft2000 - x64**
-+ Docker Hub - [https://hub.docker.com/r/technosoft2000/calibre-web/](https://hub.docker.com/r/technosoft2000/calibre-web/)
+ Docker Hub - [https://hub.docker.com/r/technosoft2000/calibre-web](https://hub.docker.com/r/technosoft2000/calibre-web)
 + Github - [https://github.com/Technosoft2000/docker-calibre-web](https://github.com/Technosoft2000/docker-calibre-web) 

    Includes the Calibre `ebook-convert` binary.
    + The "path to convertertool" should be set to `/opt/calibre/ebook-convert`

 #### **LinuxServer - x64, armhf, aarch64**
-+ Docker Hub - [https://hub.docker.com/r/linuxserver/calibre-web/](https://hub.docker.com/r/linuxserver/calibre-web/)
+ Docker Hub - [https://hub.docker.com/r/linuxserver/calibre-web](https://hub.docker.com/r/linuxserver/calibre-web)
 + Github - [https://github.com/linuxserver/docker-calibre-web](https://github.com/linuxserver/docker-calibre-web)
 + Github - (Optional Calibre layer) - [https://github.com/linuxserver/docker-calibre-web/tree/calibre](https://github.com/linuxserver/docker-calibre-web/tree/calibre) 

@@ -85,3 +83,7 @@ Pre-built Docker images are available in these Docker Hub repositories:
 # Wiki

 For further information, How To's and FAQ please check the [Wiki](https://github.com/janeczku/calibre-web/wiki)
+
+# Contributing to Calibre-Web
+
+Please have a look at our [Contributing Guidelines](https://github.com/janeczku/calibre-web/blob/master/CONTRIBUTING.md) 
@@ -37,6 +37,7 @@ from . import config_sql, logger, cache_buster, cli, ub, db
 from .reverseproxy import ReverseProxied
 from .server import WebServer

+
 mimetypes.init()
 mimetypes.add_type('application/xhtml+xml', '.xhtml')
 mimetypes.add_type('application/epub+zip', '.epub')
@@ -56,10 +57,17 @@ mimetypes.add_type('application/ogg', '.ogg')
 mimetypes.add_type('application/ogg', '.oga')

 app = Flask(__name__)
+app.config.update(
+    SESSION_COOKIE_HTTPONLY=True,
+    SESSION_COOKIE_SAMESITE='Lax',
+    REMEMBER_COOKIE_SAMESITE='Lax',  # will be available in flask-login 0.5.1 earliest
+)
+

 lm = LoginManager()
 lm.login_view = 'web.login'
 lm.anonymous_user = ub.Anonymous
+lm.session_protection = 'strong'

 ub.init_db(cli.settingspath)
 # pylint: disable=no-member
@@ -75,6 +83,8 @@ log = logger.create()

 from . import services

+calibre_db = db.CalibreDB()
+
 def create_app():
    app.wsgi_app = ReverseProxied(app.wsgi_app)
    # For python2 convert path to unicode
@@ -91,7 +101,8 @@ def create_app():
    app.secret_key = os.getenv('SECRET_KEY', config_sql.get_flask_session_key(ub.session))

    web_server.init_app(app, config)
-    db.setup_db(config)
+    calibre_db.setup_db(config, cli.settingspath)
+    calibre_db.start()

    babel.init_app(app)
    _BABEL_TRANSLATIONS.update(str(item) for item in babel.list_translations())
@@ -30,7 +30,7 @@ import babel, pytz, requests, sqlalchemy
 import werkzeug, flask, flask_login, flask_principal, jinja2
 from flask_babel import gettext as _

-from . import db, converter, uploader, server, isoLanguages, constants
+from . import db, calibre_db, converter, uploader, server, isoLanguages, constants
 from .web import render_title_template
 try:
    from flask_login import __version__ as flask_loginVersion
@@ -85,10 +85,12 @@ _VERSIONS.update(uploader.get_versions())
@about.route("/stats")
@flask_login.login_required
 def stats():
-    counter = db.session.query(db.Books).count()
-    authors = db.session.query(db.Authors).count()
-    categorys = db.session.query(db.Tags).count()
-    series = db.session.query(db.Series).count()
-    _VERSIONS['ebook converter'] = _(converter.get_version())
+    counter = calibre_db.session.query(db.Books).count()
+    authors = calibre_db.session.query(db.Authors).count()
+    categorys = calibre_db.session.query(db.Tags).count()
+    series = calibre_db.session.query(db.Series).count()
+    _VERSIONS['ebook converter'] = _(converter.get_calibre_version())
+    _VERSIONS['unrar'] = _(converter.get_unrar_version())
+    _VERSIONS['kepubify'] = _(converter.get_kepubify_version())
    return render_title_template('stats.html', bookcounter=counter, authorcounter=authors, versions=_VERSIONS,
                                 categorycounter=categorys, seriecounter=series, title=_(u"Statistics"), page="stat")
@@ -22,6 +22,7 @@

 from __future__ import division, print_function, unicode_literals
 import os
+import re
 import base64
 import json
 import time
@@ -33,12 +34,12 @@ from flask import Blueprint, flash, redirect, url_for, abort, request, make_resp
 from flask_login import login_required, current_user, logout_user
 from flask_babel import gettext as _
 from sqlalchemy import and_
-from sqlalchemy.exc import IntegrityError
+from sqlalchemy.exc import IntegrityError, OperationalError, InvalidRequestError
 from sqlalchemy.sql.expression import func

 from . import constants, logger, helper, services
-from . import db, ub, web_server, get_locale, config, updater_thread, babel, gdriveutils
-from .helper import speaking_language, check_valid_domain, send_test_mail, reset_password, generate_password_hash
+from . import db, calibre_db, ub, web_server, get_locale, config, updater_thread, babel, gdriveutils
+from .helper import check_valid_domain, send_test_mail, reset_password, generate_password_hash
 from .gdriveutils import is_gdrive_ready, gdrive_support
 from .web import admin_required, render_title_template, before_request, unconfigured, login_required_if_no_ano

@@ -85,7 +86,7 @@ def shutdown():
    showtext = {}
    if task in (0, 1):  # valid commandos received
        # close all database connections
-        db.dispose()
+        calibre_db.dispose()
        ub.dispose()

        if task == 0:
@@ -98,7 +99,7 @@ def shutdown():

    if task == 2:
        log.warning("reconnecting to calibre database")
-        db.setup_db(config)
+        calibre_db.reconnect_db(config, ub.app_DB_path)
        showtext['text'] = _(u'Reconnect successful')
        return json.dumps(showtext)

@@ -131,6 +132,7 @@ def admin():
    allUser = ub.session.query(ub.User).all()
    email_settings = config.get_mail_settings()
    return render_title_template("admin.html", allUser=allUser, email=email_settings, config=config, commit=commit,
+                                 feature_support=feature_support,
                                 title=_(u"Admin page"), page="admin")


@@ -147,10 +149,10 @@ def configuration():
@login_required
@admin_required
 def view_configuration():
-    readColumn = db.session.query(db.Custom_Columns)\
-            .filter(and_(db.Custom_Columns.datatype == 'bool',db.Custom_Columns.mark_for_delete == 0)).all()
-    restrictColumns= db.session.query(db.Custom_Columns)\
-            .filter(and_(db.Custom_Columns.datatype == 'text',db.Custom_Columns.mark_for_delete == 0)).all()
+    readColumn = calibre_db.session.query(db.Custom_Columns)\
+            .filter(and_(db.Custom_Columns.datatype == 'bool', db.Custom_Columns.mark_for_delete == 0)).all()
+    restrictColumns= calibre_db.session.query(db.Custom_Columns)\
+            .filter(and_(db.Custom_Columns.datatype == 'text', db.Custom_Columns.mark_for_delete == 0)).all()
    return render_title_template("config_view_edit.html", conf=config, readColumns=readColumn,
                                 restrictColumns=restrictColumns,
                                 title=_(u"UI Configuration"), page="uiconfig")
@@ -168,7 +170,6 @@ def update_view_configuration():

    _config_string("config_calibre_web_title")
    _config_string("config_columns_to_ignore")
-    # _config_string("config_mature_content_tags")
    reboot_required |= _config_string("config_title_regex")

    _config_int("config_read_column")
@@ -178,9 +179,6 @@ def update_view_configuration():
    _config_int("config_authors_max")
    _config_int("config_restricted_column")

-    if config.config_google_drive_watch_changes_response:
-        config.config_google_drive_watch_changes_response = json.dumps(config.config_google_drive_watch_changes_response)
-
    config.config_default_role = constants.selected_roles(to_save)
    config.config_default_role &= ~constants.ROLE_ANONYMOUS

@@ -429,7 +427,6 @@ def delete_restriction(res_type):
    return ""


-#@admi.route("/ajax/listrestriction/<int:type>/<int:user_id>", defaults={'user_id': '0'})
@admi.route("/ajax/listrestriction/<int:res_type>")
@login_required
@admin_required
@@ -475,6 +472,7 @@ def list_restriction(res_type):
    response.headers["Content-Type"] = "application/json; charset=utf-8"
    return response

+
@admi.route("/config", methods=["GET", "POST"])
@unconfigured
 def basic_configuration():
@@ -484,19 +482,23 @@ def basic_configuration():
    return _configuration_result()


-def _configuration_update_helper():
-    reboot_required = False
-    db_change = False
-    to_save = request.form.to_dict()
+def _config_int(to_save, x, func=int):
+    return config.set_from_dictionary(to_save, x, func)

-    _config_string = lambda x: config.set_from_dictionary(to_save, x, lambda y: y.strip() if y else y)
-    _config_int = lambda x: config.set_from_dictionary(to_save, x, int)
-    _config_checkbox = lambda x: config.set_from_dictionary(to_save, x, lambda y: y == "on", False)
-    _config_checkbox_int = lambda x: config.set_from_dictionary(to_save, x, lambda y: 1 if (y == "on") else 0, 0)

-    db_change |= _config_string("config_calibre_dir")
+def _config_checkbox(to_save, x):
+    return config.set_from_dictionary(to_save, x, lambda y: y == "on", False)

-    # Google drive setup
+
+def _config_checkbox_int(to_save, x):
+    return config.set_from_dictionary(to_save, x, lambda y: 1 if (y == "on") else 0, 0)
+
+
+def _config_string(to_save, x):
+    return config.set_from_dictionary(to_save, x, lambda y: y.strip() if y else y)
+
+
+def _configuration_gdrive_helper(to_save):
    if not os.path.isfile(gdriveutils.SETTINGS_YAML):
        config.config_use_google_drive = False

@@ -515,147 +517,187 @@ def _configuration_update_helper():

    # always show google drive settings, but in case of error deny support
    config.config_use_google_drive = (not gdriveError) and ("config_use_google_drive" in to_save)
-    if _config_string("config_google_drive_folder"):
+    if _config_string(to_save, "config_google_drive_folder"):
        gdriveutils.deleteDatabaseOnChange()
+    return gdriveError

-    reboot_required |= _config_int("config_port")
+def _configuration_oauth_helper(to_save):
+    active_oauths = 0
+    reboot_required = False
+    for element in oauthblueprints:
+        if to_save["config_" + str(element['id']) + "_oauth_client_id"] != element['oauth_client_id'] \
+            or to_save["config_" + str(element['id']) + "_oauth_client_secret"] != element['oauth_client_secret']:
+            reboot_required = True
+            element['oauth_client_id'] = to_save["config_" + str(element['id']) + "_oauth_client_id"]
+            element['oauth_client_secret'] = to_save["config_" + str(element['id']) + "_oauth_client_secret"]
+        if to_save["config_" + str(element['id']) + "_oauth_client_id"] \
+            and to_save["config_" + str(element['id']) + "_oauth_client_secret"]:
+            active_oauths += 1
+            element["active"] = 1
+        else:
+            element["active"] = 0
+        ub.session.query(ub.OAuthProvider).filter(ub.OAuthProvider.id == element['id']).update(
+            {"oauth_client_id": to_save["config_" + str(element['id']) + "_oauth_client_id"],
+             "oauth_client_secret": to_save["config_" + str(element['id']) + "_oauth_client_secret"],
+             "active": element["active"]})
+    return reboot_required

-    reboot_required |= _config_string("config_keyfile")
-    if config.config_keyfile and not os.path.isfile(config.config_keyfile):
-        return _configuration_result(_('Keyfile Location is not Valid, Please Enter Correct Path'), gdriveError)
-
-    reboot_required |= _config_string("config_certfile")
-    if config.config_certfile and not os.path.isfile(config.config_certfile):
-        return _configuration_result(_('Certfile Location is not Valid, Please Enter Correct Path'), gdriveError)
-
-    _config_checkbox_int("config_uploading")
-    _config_checkbox_int("config_anonbrowse")
-    _config_checkbox_int("config_public_reg")
-    reboot_required |= _config_checkbox_int("config_kobo_sync")
-    _config_checkbox_int("config_kobo_proxy")
-
-
-    _config_int("config_ebookconverter")
-    _config_string("config_calibre")
-    _config_string("config_converterpath")
-
-    reboot_required |= _config_int("config_login_type")
-
-    #LDAP configurator,
-    if config.config_login_type == constants.LOGIN_LDAP:
-        reboot_required |= _config_string("config_ldap_provider_url")
-        reboot_required |= _config_int("config_ldap_port")
-        reboot_required |= _config_int("config_ldap_authentication")
-        reboot_required |= _config_string("config_ldap_dn")
-        reboot_required |= _config_string("config_ldap_serv_username")
-        reboot_required |= _config_string("config_ldap_user_object")
-        reboot_required |= _config_string("config_ldap_group_object_filter")
-        reboot_required |= _config_string("config_ldap_group_members_field")
-        reboot_required |= _config_checkbox("config_ldap_openldap")
-        reboot_required |= _config_int("config_ldap_encryption")
-        reboot_required |= _config_string("config_ldap_cert_path")
-        _config_string("config_ldap_group_name")
-        if "config_ldap_serv_password" in to_save and to_save["config_ldap_serv_password"] != "":
-            reboot_required |= 1
-            config.set_from_dictionary(to_save, "config_ldap_serv_password", base64.b64encode, encode='UTF-8')
-        config.save()
-
-        if not config.config_ldap_provider_url \
-            or not config.config_ldap_port \
-            or not config.config_ldap_dn \
-            or not config.config_ldap_user_object:
-                return _configuration_result(_('Please Enter a LDAP Provider, '
-                                             'Port, DN and User Object Identifier'), gdriveError)
-
-        if config.config_ldap_authentication > constants.LDAP_AUTH_ANONYMOUS:
-            if config.config_ldap_authentication > constants.LDAP_AUTH_UNAUTHENTICATE:
-                if not config.config_ldap_serv_username or not bool(config.config_ldap_serv_password):
-                    return _configuration_result('Please Enter a LDAP Service Account and Password', gdriveError)
-            else:
-                if not config.config_ldap_serv_username:
-                    return _configuration_result('Please Enter a LDAP Service Account', gdriveError)
-
-        #_config_checkbox("config_ldap_use_ssl")
-        #_config_checkbox("config_ldap_use_tls")
-        # reboot_required |= _config_checkbox("config_ldap_openldap")
-        # _config_checkbox("config_ldap_require_cert")
-
-        if config.config_ldap_group_object_filter:
-            if config.config_ldap_group_object_filter.count("%s") != 1:
-                return _configuration_result(_('LDAP Group Object Filter Needs to Have One "%s" Format Identifier'),
-                                             gdriveError)
-            if config.config_ldap_group_object_filter.count("(") != config.config_ldap_group_object_filter.count(")"):
-                return _configuration_result(_('LDAP Group Object Filter Has Unmatched Parenthesis'),
-                                             gdriveError)
-
-        if config.config_ldap_user_object.count("%s") != 1:
-            return _configuration_result(_('LDAP User Object Filter needs to Have One "%s" Format Identifier'),
-                                         gdriveError)
-        if config.config_ldap_user_object.count("(") != config.config_ldap_user_object.count(")"):
-            return _configuration_result(_('LDAP User Object Filter Has Unmatched Parenthesis'),
-                                         gdriveError)
-
-        if config.config_ldap_cert_path and not os.path.isdir(config.config_ldap_cert_path):
-            return _configuration_result(_('LDAP Certificate Location is not Valid, Please Enter Correct Path'),
-                                         gdriveError)
-
-    # Remote login configuration
-    _config_checkbox("config_remote_login")
-    if not config.config_remote_login:
-        ub.session.query(ub.RemoteAuthToken).filter(ub.RemoteAuthToken.token_type==0).delete()
-
-    # Goodreads configuration
-    _config_checkbox("config_use_goodreads")
-    _config_string("config_goodreads_api_key")
-    _config_string("config_goodreads_api_secret")
-    if services.goodreads_support:
-        services.goodreads_support.connect(config.config_goodreads_api_key,
-                                           config.config_goodreads_api_secret,
-                                           config.config_use_goodreads)
-
-    _config_int("config_updatechannel")
-
-    # Reverse proxy login configuration
-    _config_checkbox("config_allow_reverse_proxy_header_login")
-    _config_string("config_reverse_proxy_login_header_name")
-
-    # GitHub OAuth configuration
-    if config.config_login_type == constants.LOGIN_OAUTH:
-        active_oauths = 0
-
-        for element in oauthblueprints:
-            if to_save["config_" + str(element['id']) + "_oauth_client_id"] != element['oauth_client_id'] \
-                or to_save["config_" + str(element['id']) + "_oauth_client_secret"] != element['oauth_client_secret']:
-                reboot_required = True
-                element['oauth_client_id'] = to_save["config_" + str(element['id']) + "_oauth_client_id"]
-                element['oauth_client_secret'] = to_save["config_" + str(element['id']) + "_oauth_client_secret"]
-            if to_save["config_"+str(element['id'])+"_oauth_client_id"] \
-               and to_save["config_"+str(element['id'])+"_oauth_client_secret"]:
-                active_oauths += 1
-                element["active"] = 1
-            else:
-                element["active"] = 0
-            ub.session.query(ub.OAuthProvider).filter(ub.OAuthProvider.id == element['id']).update(
-                {"oauth_client_id":to_save["config_"+str(element['id'])+"_oauth_client_id"],
-                "oauth_client_secret":to_save["config_"+str(element['id'])+"_oauth_client_secret"],
-                "active":element["active"]})
-
-
-    reboot_required |= _config_int("config_log_level")
-    reboot_required |= _config_string("config_logfile")
+def _configuration_logfile_helper(to_save, gdriveError):
+    reboot_required = False
+    reboot_required |= _config_int(to_save, "config_log_level")
+    reboot_required |= _config_string(to_save, "config_logfile")
    if not logger.is_valid_logfile(config.config_logfile):
-        return _configuration_result(_('Logfile Location is not Valid, Please Enter Correct Path'), gdriveError)
+        return reboot_required, _configuration_result(_('Logfile Location is not Valid, Please Enter Correct Path'), gdriveError)

-    reboot_required |= _config_checkbox_int("config_access_log")
-    reboot_required |= _config_string("config_access_logfile")
+    reboot_required |= _config_checkbox_int(to_save, "config_access_log")
+    reboot_required |= _config_string(to_save, "config_access_logfile")
    if not logger.is_valid_logfile(config.config_access_logfile):
-        return _configuration_result(_('Access Logfile Location is not Valid, Please Enter Correct Path'), gdriveError)
+        return reboot_required, _configuration_result(_('Access Logfile Location is not Valid, Please Enter Correct Path'), gdriveError)
+    return reboot_required, None

-    # Rarfile Content configuration
-    _config_string("config_rarfile_location")
-    unrar_status = helper.check_unrar(config.config_rarfile_location)
-    if unrar_status:
-        return _configuration_result(unrar_status, gdriveError)
+def _configuration_ldap_helper(to_save, gdriveError):
+    reboot_required = False
+    reboot_required |= _config_string(to_save, "config_ldap_provider_url")
+    reboot_required |= _config_int(to_save, "config_ldap_port")
+    reboot_required |= _config_int(to_save, "config_ldap_authentication")
+    reboot_required |= _config_string(to_save, "config_ldap_dn")
+    reboot_required |= _config_string(to_save, "config_ldap_serv_username")
+    reboot_required |= _config_string(to_save, "config_ldap_user_object")
+    reboot_required |= _config_string(to_save, "config_ldap_group_object_filter")
+    reboot_required |= _config_string(to_save, "config_ldap_group_members_field")
+    reboot_required |= _config_checkbox(to_save, "config_ldap_openldap")
+    reboot_required |= _config_int(to_save, "config_ldap_encryption")
+    reboot_required |= _config_string(to_save, "config_ldap_cert_path")
+    _config_string(to_save, "config_ldap_group_name")
+    if "config_ldap_serv_password" in to_save and to_save["config_ldap_serv_password"] != "":
+        reboot_required |= 1
+        config.set_from_dictionary(to_save, "config_ldap_serv_password", base64.b64encode, encode='UTF-8')
+    config.save()
+
+    if not config.config_ldap_provider_url \
+        or not config.config_ldap_port \
+        or not config.config_ldap_dn \
+        or not config.config_ldap_user_object:
+        return reboot_required, _configuration_result(_('Please Enter a LDAP Provider, '
+                                       'Port, DN and User Object Identifier'), gdriveError)
+
+    if config.config_ldap_authentication > constants.LDAP_AUTH_ANONYMOUS:
+        if config.config_ldap_authentication > constants.LDAP_AUTH_UNAUTHENTICATE:
+            if not config.config_ldap_serv_username or not bool(config.config_ldap_serv_password):
+                return reboot_required, _configuration_result('Please Enter a LDAP Service Account and Password', gdriveError)
+        else:
+            if not config.config_ldap_serv_username:
+                return reboot_required, _configuration_result('Please Enter a LDAP Service Account', gdriveError)
+
+    if config.config_ldap_group_object_filter:
+        if config.config_ldap_group_object_filter.count("%s") != 1:
+            return reboot_required, _configuration_result(_('LDAP Group Object Filter Needs to Have One "%s" Format Identifier'),
+                                         gdriveError)
+        if config.config_ldap_group_object_filter.count("(") != config.config_ldap_group_object_filter.count(")"):
+            return reboot_required, _configuration_result(_('LDAP Group Object Filter Has Unmatched Parenthesis'),
+                                         gdriveError)
+
+    if config.config_ldap_user_object.count("%s") != 1:
+        return reboot_required, _configuration_result(_('LDAP User Object Filter needs to Have One "%s" Format Identifier'),
+                                     gdriveError)
+    if config.config_ldap_user_object.count("(") != config.config_ldap_user_object.count(")"):
+        return reboot_required, _configuration_result(_('LDAP User Object Filter Has Unmatched Parenthesis'),
+                                     gdriveError)
+
+    if config.config_ldap_cert_path and not os.path.isdir(config.config_ldap_cert_path):
+        return reboot_required, _configuration_result(_('LDAP Certificate Location is not Valid, Please Enter Correct Path'),
+                                     gdriveError)
+    return reboot_required, None
+
+
+def _configuration_update_helper():
+    reboot_required = False
+    db_change = False
+    to_save = request.form.to_dict()
+    gdriveError = None
+
+    to_save['config_calibre_dir'] = re.sub('[\\/]metadata\.db$', '', to_save['config_calibre_dir'], flags=re.IGNORECASE)
+    try:
+        db_change |= _config_string(to_save, "config_calibre_dir")
+
+        # Google drive setup
+        gdriveError = _configuration_gdrive_helper(to_save)
+
+        reboot_required |= _config_int(to_save, "config_port")
+
+        reboot_required |= _config_string(to_save, "config_keyfile")
+        if config.config_keyfile and not os.path.isfile(config.config_keyfile):
+            return _configuration_result(_('Keyfile Location is not Valid, Please Enter Correct Path'), gdriveError)
+
+        reboot_required |= _config_string(to_save, "config_certfile")
+        if config.config_certfile and not os.path.isfile(config.config_certfile):
+            return _configuration_result(_('Certfile Location is not Valid, Please Enter Correct Path'), gdriveError)
+
+        _config_checkbox_int(to_save, "config_uploading")
+        _config_checkbox_int(to_save, "config_anonbrowse")
+        _config_checkbox_int(to_save, "config_public_reg")
+        _config_checkbox_int(to_save, "config_register_email")
+        reboot_required |= _config_checkbox_int(to_save, "config_kobo_sync")
+        _config_int(to_save, "config_external_port")
+        _config_checkbox_int(to_save, "config_kobo_proxy")
+
+        if "config_upload_formats" in to_save:
+            to_save["config_upload_formats"] = ','.join(
+                helper.uniq([x.lstrip().rstrip().lower() for x in to_save["config_upload_formats"].split(',')]))
+            _config_string(to_save, "config_upload_formats")
+            constants.EXTENSIONS_UPLOAD = config.config_upload_formats.split(',')
+
+        _config_string(to_save, "config_calibre")
+        _config_string(to_save, "config_converterpath")
+        _config_string(to_save, "config_kepubifypath")
+
+        reboot_required |= _config_int(to_save, "config_login_type")
+
+        #LDAP configurator,
+        if config.config_login_type == constants.LOGIN_LDAP:
+            reboot, message = _configuration_ldap_helper(to_save, gdriveError)
+            if message:
+                return message
+            reboot_required |= reboot
+
+        # Remote login configuration
+
+        _config_checkbox(to_save, "config_remote_login")
+        if not config.config_remote_login:
+            ub.session.query(ub.RemoteAuthToken).filter(ub.RemoteAuthToken.token_type==0).delete()
+
+        # Goodreads configuration
+        _config_checkbox(to_save, "config_use_goodreads")
+        _config_string(to_save, "config_goodreads_api_key")
+        _config_string(to_save, "config_goodreads_api_secret")
+        if services.goodreads_support:
+            services.goodreads_support.connect(config.config_goodreads_api_key,
+                                               config.config_goodreads_api_secret,
+                                               config.config_use_goodreads)
+
+        _config_int(to_save, "config_updatechannel")
+
+        # Reverse proxy login configuration
+        _config_checkbox(to_save, "config_allow_reverse_proxy_header_login")
+        _config_string(to_save, "config_reverse_proxy_login_header_name")
+
+        # OAuth configuration
+        if config.config_login_type == constants.LOGIN_OAUTH:
+            reboot_required |= _configuration_oauth_helper(to_save)
+
+        reboot, message = _configuration_logfile_helper(to_save, gdriveError)
+        if message:
+            return message
+        reboot_required |= reboot
+        # Rarfile Content configuration
+        _config_string(to_save, "config_rarfile_location")
+        if "config_rarfile_location" in to_save:
+            unrar_status = helper.check_unrar(config.config_rarfile_location)
+            if unrar_status:
+                return _configuration_result(unrar_status, gdriveError)
+    except (OperationalError, InvalidRequestError):
+        ub.session.rollback()
+        _configuration_result(_(u"Settings DB is not Writeable"), gdriveError)

    try:
        metadata_db = os.path.join(config.config_calibre_dir, "metadata.db")
@@ -666,9 +708,10 @@ def _configuration_update_helper():
        return _configuration_result('%s' % e, gdriveError)

    if db_change:
-        # reload(db)
-        if not db.setup_db(config):
+        if not calibre_db.setup_db(config, ub.app_DB_path):
            return _configuration_result(_('DB Location is not Valid, Please Enter Correct Path'), gdriveError)
+        if not os.access(os.path.join(config.config_calibre_dir, "metadata.db"), os.W_OK):
+            flash(_(u"DB is not Writeable"), category="warning")

    config.save()
    flash(_(u"Calibre-Web configuration updated"), category="success")
@@ -687,7 +730,7 @@ def _configuration_result(error_flash=None, gdriveError=None):
        gdriveError = _(gdriveError)
    else:
        # if config.config_use_google_drive and\
-        if not gdrive_authenticate:
+        if not gdrive_authenticate and gdrive_support:
            gdrivefolders = gdriveutils.listRootFolders()

    show_back_button = current_user.is_authenticated
@@ -704,62 +747,161 @@ def _configuration_result(error_flash=None, gdriveError=None):
                                 title=_(u"Basic Configuration"), page="config")


+def _handle_new_user(to_save, content,languages, translations, kobo_support):
+    content.default_language = to_save["default_language"]
+    # content.mature_content = "Show_mature_content" in to_save
+    content.locale = to_save.get("locale", content.locale)
+
+    content.sidebar_view = sum(int(key[5:]) for key in to_save if key.startswith('show_'))
+    if "show_detail_random" in to_save:
+        content.sidebar_view |= constants.DETAIL_RANDOM
+
+    content.role = constants.selected_roles(to_save)
+
+    if not to_save["nickname"] or not to_save["email"] or not to_save["password"]:
+        flash(_(u"Please fill out all fields!"), category="error")
+        return render_title_template("user_edit.html", new_user=1, content=content, translations=translations,
+                                     registered_oauth=oauth_check, kobo_support=kobo_support,
+                                     title=_(u"Add new user"))
+    content.password = generate_password_hash(to_save["password"])
+    existing_user = ub.session.query(ub.User).filter(func.lower(ub.User.nickname) == to_save["nickname"].lower()) \
+        .first()
+    existing_email = ub.session.query(ub.User).filter(ub.User.email == to_save["email"].lower()) \
+        .first()
+    if not existing_user and not existing_email:
+        content.nickname = to_save["nickname"]
+        if config.config_public_reg and not check_valid_domain(to_save["email"]):
+            flash(_(u"E-mail is not from valid domain"), category="error")
+            return render_title_template("user_edit.html", new_user=1, content=content, translations=translations,
+                                         registered_oauth=oauth_check, kobo_support=kobo_support,
+                                         title=_(u"Add new user"))
+        else:
+            content.email = to_save["email"]
+    else:
+        flash(_(u"Found an existing account for this e-mail address or nickname."), category="error")
+        return render_title_template("user_edit.html", new_user=1, content=content, translations=translations,
+                                     languages=languages, title=_(u"Add new user"), page="newuser",
+                                     kobo_support=kobo_support, registered_oauth=oauth_check)
+    try:
+        content.allowed_tags = config.config_allowed_tags
+        content.denied_tags = config.config_denied_tags
+        content.allowed_column_value = config.config_allowed_column_value
+        content.denied_column_value = config.config_denied_column_value
+        ub.session.add(content)
+        ub.session.commit()
+        flash(_(u"User '%(user)s' created", user=content.nickname), category="success")
+        return redirect(url_for('admin.admin'))
+    except IntegrityError:
+        ub.session.rollback()
+        flash(_(u"Found an existing account for this e-mail address or nickname."), category="error")
+    except OperationalError:
+        ub.session.rollback()
+        flash(_(u"Settings DB is not Writeable"), category="error")
+
+
+def _handle_edit_user(to_save, content,languages, translations, kobo_support, downloads):
+    if "delete" in to_save:
+        if ub.session.query(ub.User).filter(ub.User.role.op('&')(constants.ROLE_ADMIN) == constants.ROLE_ADMIN,
+                                            ub.User.id != content.id).count():
+            ub.session.query(ub.User).filter(ub.User.id == content.id).delete()
+            ub.session.commit()
+            flash(_(u"User '%(nick)s' deleted", nick=content.nickname), category="success")
+            return redirect(url_for('admin.admin'))
+        else:
+            flash(_(u"No admin user remaining, can't delete user", nick=content.nickname), category="error")
+            return redirect(url_for('admin.admin'))
+    else:
+        if not ub.session.query(ub.User).filter(ub.User.role.op('&')(constants.ROLE_ADMIN) == constants.ROLE_ADMIN,
+                                                ub.User.id != content.id).count() and \
+            not 'admin_role' in to_save:
+            flash(_(u"No admin user remaining, can't remove admin role", nick=content.nickname), category="error")
+            return redirect(url_for('admin.admin'))
+
+        if "password" in to_save and to_save["password"]:
+            content.password = generate_password_hash(to_save["password"])
+        anonymous = content.is_anonymous
+        content.role = constants.selected_roles(to_save)
+        if anonymous:
+            content.role |= constants.ROLE_ANONYMOUS
+        else:
+            content.role &= ~constants.ROLE_ANONYMOUS
+
+        val = [int(k[5:]) for k in to_save if k.startswith('show_')]
+        sidebar = ub.get_sidebar_config()
+        for element in sidebar:
+            value = element['visibility']
+            if value in val and not content.check_visibility(value):
+                content.sidebar_view |= value
+            elif not value in val and content.check_visibility(value):
+                content.sidebar_view &= ~value
+
+        if "Show_detail_random" in to_save:
+            content.sidebar_view |= constants.DETAIL_RANDOM
+        else:
+            content.sidebar_view &= ~constants.DETAIL_RANDOM
+
+        if "default_language" in to_save:
+            content.default_language = to_save["default_language"]
+        if "locale" in to_save and to_save["locale"]:
+            content.locale = to_save["locale"]
+        if to_save["email"] and to_save["email"] != content.email:
+            existing_email = ub.session.query(ub.User).filter(ub.User.email == to_save["email"].lower()) \
+                .first()
+            if not existing_email:
+                content.email = to_save["email"]
+            else:
+                flash(_(u"Found an existing account for this e-mail address."), category="error")
+                return render_title_template("user_edit.html",
+                                             translations=translations,
+                                             languages=languages,
+                                             mail_configured=config.get_mail_server_configured(),
+                                             kobo_support=kobo_support,
+                                             new_user=0,
+                                             content=content,
+                                             downloads=downloads,
+                                             registered_oauth=oauth_check,
+                                             title=_(u"Edit User %(nick)s", nick=content.nickname), page="edituser")
+        if "nickname" in to_save and to_save["nickname"] != content.nickname:
+            # Query User nickname, if not existing, change
+            if not ub.session.query(ub.User).filter(ub.User.nickname == to_save["nickname"]).scalar():
+                content.nickname = to_save["nickname"]
+            else:
+                flash(_(u"This username is already taken"), category="error")
+                return render_title_template("user_edit.html",
+                                             translations=translations,
+                                             languages=languages,
+                                             mail_configured=config.get_mail_server_configured(),
+                                             new_user=0, content=content,
+                                             downloads=downloads,
+                                             registered_oauth=oauth_check,
+                                             kobo_support=kobo_support,
+                                             title=_(u"Edit User %(nick)s", nick=content.nickname),
+                                             page="edituser")
+
+        if "kindle_mail" in to_save and to_save["kindle_mail"] != content.kindle_mail:
+            content.kindle_mail = to_save["kindle_mail"]
+    try:
+        ub.session.commit()
+        flash(_(u"User '%(nick)s' updated", nick=content.nickname), category="success")
+    except IntegrityError:
+        ub.session.rollback()
+        flash(_(u"An unknown error occured."), category="error")
+    except OperationalError:
+        ub.session.rollback()
+        flash(_(u"Settings DB is not Writeable"), category="error")
+
+
@admi.route("/admin/user/new", methods=["GET", "POST"])
@login_required
@admin_required
 def new_user():
    content = ub.User()
-    languages = speaking_language()
+    languages = calibre_db.speaking_language()
    translations = [LC('en')] + babel.list_translations()
    kobo_support = feature_support['kobo'] and config.config_kobo_sync
    if request.method == "POST":
        to_save = request.form.to_dict()
-        content.default_language = to_save["default_language"]
-        # content.mature_content = "Show_mature_content" in to_save
-        content.locale = to_save.get("locale", content.locale)
-
-        content.sidebar_view = sum(int(key[5:]) for key in to_save if key.startswith('show_'))
-        if "show_detail_random" in to_save:
-            content.sidebar_view |= constants.DETAIL_RANDOM
-
-        content.role = constants.selected_roles(to_save)
-
-        if not to_save["nickname"] or not to_save["email"] or not to_save["password"]:
-            flash(_(u"Please fill out all fields!"), category="error")
-            return render_title_template("user_edit.html", new_user=1, content=content, translations=translations,
-                                         registered_oauth=oauth_check, kobo_support=kobo_support,
-                                         title=_(u"Add new user"))
-        content.password = generate_password_hash(to_save["password"])
-        existing_user = ub.session.query(ub.User).filter(func.lower(ub.User.nickname) == to_save["nickname"].lower())\
-            .first()
-        existing_email = ub.session.query(ub.User).filter(ub.User.email == to_save["email"].lower())\
-            .first()
-        if not existing_user and not existing_email:
-            content.nickname = to_save["nickname"]
-            if config.config_public_reg and not check_valid_domain(to_save["email"]):
-                flash(_(u"E-mail is not from valid domain"), category="error")
-                return render_title_template("user_edit.html", new_user=1, content=content, translations=translations,
-                                             registered_oauth=oauth_check, kobo_support=kobo_support,
-                                             title=_(u"Add new user"))
-            else:
-                content.email = to_save["email"]
-        else:
-            flash(_(u"Found an existing account for this e-mail address or nickname."), category="error")
-            return render_title_template("user_edit.html", new_user=1, content=content, translations=translations,
-                                     languages=languages, title=_(u"Add new user"), page="newuser",
-                                     kobo_support=kobo_support, registered_oauth=oauth_check)
-        try:
-            content.allowed_tags = config.config_allowed_tags
-            content.denied_tags = config.config_denied_tags
-            content.allowed_column_value = config.config_allowed_column_value
-            content.denied_column_value = config.config_denied_column_value
-            ub.session.add(content)
-            ub.session.commit()
-            flash(_(u"User '%(user)s' created", user=content.nickname), category="success")
-            return redirect(url_for('admin.admin'))
-        except IntegrityError:
-            ub.session.rollback()
-            flash(_(u"Found an existing account for this e-mail address or nickname."), category="error")
+        _handle_new_user(to_save, content, languages, translations, kobo_support)
    else:
        content.role = config.config_default_role
        content.sidebar_view = config.config_default_show
@@ -773,8 +915,7 @@ def new_user():
@admin_required
 def edit_mailsettings():
    content = config.get_mail_settings()
-    # log.debug("edit_mailsettings %r", content)
-    return render_title_template("email_edit.html", content=content, title=_(u"Edit e-mail server settings"),
+    return render_title_template("email_edit.html", content=content, title=_(u"Edit E-mail Server Settings"),
                                 page="mailset")


@@ -783,18 +924,21 @@ def edit_mailsettings():
@admin_required
 def update_mailsettings():
    to_save = request.form.to_dict()
-    log.debug("update_mailsettings %r", to_save)
+    # log.debug("update_mailsettings %r", to_save)

-    _config_string = lambda x: config.set_from_dictionary(to_save, x, lambda y: y.strip() if y else y)
-    _config_int = lambda x: config.set_from_dictionary(to_save, x, int)
-
-    _config_string("mail_server")
-    _config_int("mail_port")
-    _config_int("mail_use_ssl")
-    _config_string("mail_login")
-    _config_string("mail_password")
-    _config_string("mail_from")
-    config.save()
+    _config_string(to_save, "mail_server")
+    _config_int(to_save, "mail_port")
+    _config_int(to_save, "mail_use_ssl")
+    _config_string(to_save, "mail_login")
+    _config_string(to_save, "mail_password")
+    _config_string(to_save, "mail_from")
+    _config_int(to_save, "mail_size", lambda y: int(y)*1024*1024)
+    try:
+        config.save()
+    except (OperationalError, InvalidRequestError):
+        ub.session.rollback()
+        flash(_(u"Settings DB is not Writeable"), category="error")
+        return edit_mailsettings()

    if to_save.get("test"):
        if current_user.email:
@@ -821,105 +965,18 @@ def edit_user(user_id):
        flash(_(u"User not found"), category="error")
        return redirect(url_for('admin.admin'))
    downloads = list()
-    languages = speaking_language()
+    languages = calibre_db.speaking_language()
    translations = babel.list_translations() + [LC('en')]
    kobo_support = feature_support['kobo'] and config.config_kobo_sync
    for book in content.downloads:
-        downloadbook = db.session.query(db.Books).filter(db.Books.id == book.book_id).first()
+        downloadbook = calibre_db.get_book(book.book_id)
        if downloadbook:
            downloads.append(downloadbook)
        else:
            ub.delete_download(book.book_id)
-            # ub.session.query(ub.Downloads).filter(book.book_id == ub.Downloads.book_id).delete()
-            # ub.session.commit()
    if request.method == "POST":
        to_save = request.form.to_dict()
-        if "delete" in to_save:
-            if ub.session.query(ub.User).filter(ub.User.role.op('&')(constants.ROLE_ADMIN) == constants.ROLE_ADMIN,
-                                                ub.User.id != content.id).count():
-                ub.session.query(ub.User).filter(ub.User.id == content.id).delete()
-                ub.session.commit()
-                flash(_(u"User '%(nick)s' deleted", nick=content.nickname), category="success")
-                return redirect(url_for('admin.admin'))
-            else:
-                flash(_(u"No admin user remaining, can't delete user", nick=content.nickname), category="error")
-                return redirect(url_for('admin.admin'))
-        else:
-            if not ub.session.query(ub.User).filter(ub.User.role.op('&')(constants.ROLE_ADMIN) == constants.ROLE_ADMIN,
-                                                    ub.User.id != content.id).count() and \
-                not 'admin_role' in to_save:
-                flash(_(u"No admin user remaining, can't remove admin role", nick=content.nickname), category="error")
-                return redirect(url_for('admin.admin'))
-
-            if "password" in to_save and to_save["password"]:
-                content.password = generate_password_hash(to_save["password"])
-            anonymous = content.is_anonymous
-            content.role = constants.selected_roles(to_save)
-            if anonymous:
-                content.role |= constants.ROLE_ANONYMOUS
-            else:
-                content.role &= ~constants.ROLE_ANONYMOUS
-
-            val = [int(k[5:]) for k in to_save if k.startswith('show_')]
-            sidebar = ub.get_sidebar_config()
-            for element in sidebar:
-                value = element['visibility']
-                if value in val and not content.check_visibility(value):
-                    content.sidebar_view |= value
-                elif not value in val and content.check_visibility(value):
-                    content.sidebar_view &= ~value
-
-            if "Show_detail_random" in to_save:
-                content.sidebar_view |= constants.DETAIL_RANDOM
-            else:
-                content.sidebar_view &= ~constants.DETAIL_RANDOM
-
-            if "default_language" in to_save:
-                content.default_language = to_save["default_language"]
-            if "locale" in to_save and to_save["locale"]:
-                content.locale = to_save["locale"]
-            if to_save["email"] and to_save["email"] != content.email:
-                existing_email = ub.session.query(ub.User).filter(ub.User.email == to_save["email"].lower()) \
-                    .first()
-                if not existing_email:
-                    content.email = to_save["email"]
-                else:
-                    flash(_(u"Found an existing account for this e-mail address."), category="error")
-                    return render_title_template("user_edit.html",
-                                                 translations=translations,
-                                                 languages=languages,
-                                                 mail_configured = config.get_mail_server_configured(),
-                                                 kobo_support=kobo_support,
-                                                 new_user=0,
-                                                 content=content,
-                                                 downloads=downloads,
-                                                 registered_oauth=oauth_check,
-                                                 title=_(u"Edit User %(nick)s", nick=content.nickname), page="edituser")
-            if "nickname" in to_save and to_save["nickname"] != content.nickname:
-                # Query User nickname, if not existing, change
-                if not ub.session.query(ub.User).filter(ub.User.nickname == to_save["nickname"]).scalar():
-                    content.nickname = to_save["nickname"]
-                else:
-                    flash(_(u"This username is already taken"), category="error")
-                    return render_title_template("user_edit.html",
-                                                 translations=translations,
-                                                 languages=languages,
-                                                 mail_configured=config.get_mail_server_configured(),
-                                                 new_user=0, content=content,
-                                                 downloads=downloads,
-                                                 registered_oauth=oauth_check,
-                                                 kobo_support=kobo_support,
-                                                 title=_(u"Edit User %(nick)s", nick=content.nickname),
-                                                 page="edituser")
-
-            if "kindle_mail" in to_save and to_save["kindle_mail"] != content.kindle_mail:
-                content.kindle_mail = to_save["kindle_mail"]
-        try:
-            ub.session.commit()
-            flash(_(u"User '%(nick)s' updated", nick=content.nickname), category="success")
-        except IntegrityError:
-            ub.session.rollback()
-            flash(_(u"An unknown error occured."), category="error")
+        _handle_edit_user(to_save, content, languages, translations, kobo_support, downloads)
    return render_title_template("user_edit.html",
                                 translations=translations,
                                 languages=languages,
@@ -939,7 +996,7 @@ def reset_user_password(user_id):
    if current_user is not None and current_user.is_authenticated:
        ret, message = reset_password(user_id)
        if ret == 1:
-            log.debug(u"Password for user %(user)s reset", user=message)
+            log.debug(u"Password for user %s reset", message)
            flash(_(u"Password for user %(user)s reset", user=message), category="success")
        elif ret == 0:
            log.error(u"An unknown error occurred. Please try again later.")
@@ -18,10 +18,17 @@

 from __future__ import division, print_function, unicode_literals
 import os
+import io

 from . import logger, isoLanguages
 from .constants import BookMeta

+try:
+    from PIL import Image as PILImage
+    use_PIL = True
+except ImportError as e:
+    use_PIL = False
+

 log = logger.create()

@@ -29,28 +36,53 @@ log = logger.create()
 try:
    from comicapi.comicarchive import ComicArchive, MetaDataStyle
    use_comic_meta = True
-except ImportError as e:
-    log.debug('cannot import comicapi, extracting comic metadata will not work: %s', e)
+    try:
+        from comicapi import __version__ as comic_version
+    except (ImportError):
+        comic_version = ''
+except (ImportError, LookupError) as e:
+    log.debug('Cannot import comicapi, extracting comic metadata will not work: %s', e)
    import zipfile
    import tarfile
    try:
        import rarfile
        use_rarfile = True
    except ImportError as e:
-        log.debug('cannot import rarfile, extracting cover files from rar files will not work: %s', e)
+        log.debug('Cannot import rarfile, extracting cover files from rar files will not work: %s', e)
        use_rarfile = False
    use_comic_meta = False

+def _cover_processing(tmp_file_name, img, extension):
+    if use_PIL:
+        # convert to jpg because calibre only supports jpg
+        if extension in ('.png',  '.webp'):
+            imgc = PILImage.open(io.BytesIO(img))
+            im = imgc.convert('RGB')
+            tmp_bytesio = io.BytesIO()
+            im.save(tmp_bytesio, format='JPEG')
+            img = tmp_bytesio.getvalue()

-def _extractCover(tmp_file_name, original_file_extension, rarExceutable):
+    prefix = os.path.dirname(tmp_file_name)
+    if img:
+        tmp_cover_name = prefix + '/cover.jpg'
+        image = open(tmp_cover_name, 'wb')
+        image.write(img)
+        image.close()
+    else:
+        tmp_cover_name = None
+    return tmp_cover_name
+
+
+
+def _extractCover(tmp_file_name, original_file_extension, rarExecutable):
    cover_data = extension = None
    if use_comic_meta:
-        archive = ComicArchive(tmp_file_name)
+        archive = ComicArchive(tmp_file_name, rar_exe_path=rarExecutable)
        for index, name in enumerate(archive.getPageNameList()):
            ext = os.path.splitext(name)
            if len(ext) > 1:
                extension = ext[1].lower()
-                if extension == '.jpg' or extension == '.jpeg':
+                if extension in ('.jpg', '.jpeg', '.png', '.webp'):
                    cover_data = archive.getPage(index)
                    break
    else:
@@ -60,7 +92,7 @@ def _extractCover(tmp_file_name, original_file_extension, rarExceutable):
                ext = os.path.splitext(name)
                if len(ext) > 1:
                    extension = ext[1].lower()
-                    if extension == '.jpg' or extension == '.jpeg':
+                    if extension in ('.jpg', '.jpeg', '.png', '.webp'):
                        cover_data = cf.read(name)
                        break
        elif original_file_extension.upper() == '.CBT':
@@ -69,37 +101,28 @@ def _extractCover(tmp_file_name, original_file_extension, rarExceutable):
                ext = os.path.splitext(name)
                if len(ext) > 1:
                    extension = ext[1].lower()
-                    if extension == '.jpg' or extension == '.jpeg':
+                    if extension in ('.jpg', '.jpeg', '.png', '.webp'):
                        cover_data = cf.extractfile(name).read()
                        break
        elif original_file_extension.upper() == '.CBR' and use_rarfile:
            try:
-                rarfile.UNRAR_TOOL = rarExceutable
+                rarfile.UNRAR_TOOL = rarExecutable
                cf = rarfile.RarFile(tmp_file_name)
                for name in cf.getnames():
                    ext = os.path.splitext(name)
                    if len(ext) > 1:
                        extension = ext[1].lower()
-                        if extension == '.jpg' or extension == '.jpeg':
+                        if extension in ('.jpg', '.jpeg', '.png', '.webp'):
                            cover_data = cf.read(name)
                            break
            except Exception as e:
                log.debug('Rarfile failed with error: %s', e)
-
-    prefix = os.path.dirname(tmp_file_name)
-    if cover_data:
-        tmp_cover_name = prefix + '/cover' + extension
-        image = open(tmp_cover_name, 'wb')
-        image.write(cover_data)
-        image.close()
-    else:
-        tmp_cover_name = None
-    return tmp_cover_name
+    return _cover_processing(tmp_file_name, cover_data, extension)


-def get_comic_info(tmp_file_path, original_file_name, original_file_extension, rarExceutable):
+def get_comic_info(tmp_file_path, original_file_name, original_file_extension, rarExecutable):
    if use_comic_meta:
-        archive = ComicArchive(tmp_file_path, rar_exe_path=rarExceutable)
+        archive = ComicArchive(tmp_file_path, rar_exe_path=rarExecutable)
        if archive.seemsToBeAComicArchive():
            if archive.hasMetadata(MetaDataStyle.CIX):
                style = MetaDataStyle.CIX
@@ -111,21 +134,15 @@ def get_comic_info(tmp_file_path, original_file_name, original_file_extension, r
            # if style is not None:
            loadedMetadata = archive.readMetadata(style)

-            lang = loadedMetadata.language
-            if lang:
-                if len(lang) == 2:
-                     loadedMetadata.language = isoLanguages.get(part1=lang).name
-                elif len(lang) == 3:
-                     loadedMetadata.language = isoLanguages.get(part3=lang).name
-            else:
-                 loadedMetadata.language = ""
+            lang = loadedMetadata.language or ""
+            loadedMetadata.language = isoLanguages.get_lang3(lang)

            return BookMeta(
                file_path=tmp_file_path,
                extension=original_file_extension,
                title=loadedMetadata.title or original_file_name,
                author=" & ".join([credit["person"] for credit in loadedMetadata.credits if credit["role"] == "Writer"]) or u'Unknown',
-                cover=_extractCover(tmp_file_path, original_file_extension, rarExceutable),
+                cover=_extractCover(tmp_file_path, original_file_extension, rarExecutable),
                description=loadedMetadata.comments or "",
                tags="",
                series=loadedMetadata.series or "",
@@ -137,7 +154,7 @@ def get_comic_info(tmp_file_path, original_file_name, original_file_extension, r
        extension=original_file_extension,
        title=original_file_name,
        author=u'Unknown',
-        cover=_extractCover(tmp_file_path, original_file_extension, rarExceutable),
+        cover=_extractCover(tmp_file_path, original_file_extension, rarExecutable),
        description="",
        tags="",
        series="",
@@ -22,7 +22,7 @@ import os
 import json
 import sys

-from sqlalchemy import exc, Column, String, Integer, SmallInteger, Boolean, BLOB
+from sqlalchemy import exc, Column, String, Integer, SmallInteger, Boolean, BLOB, JSON
 from sqlalchemy.ext.declarative import declarative_base

 from . import constants, cli, logger, ub
@@ -53,9 +53,11 @@ class _Settings(_Base):
    mail_login = Column(String, default='mail@example.com')
    mail_password = Column(String, default='mypassword')
    mail_from = Column(String, default='automailer <mail@example.com>')
+    mail_size = Column(Integer, default=25*1024*1024)

    config_calibre_dir = Column(String)
    config_port = Column(Integer, default=constants.DEFAULT_PORT)
+    config_external_port = Column(Integer, default=constants.DEFAULT_PORT)
    config_certfile = Column(String)
    config_keyfile = Column(String)

@@ -91,17 +93,16 @@ class _Settings(_Base):

    config_use_google_drive = Column(Boolean, default=False)
    config_google_drive_folder = Column(String)
-    config_google_drive_watch_changes_response = Column(String)
+    config_google_drive_watch_changes_response = Column(JSON, default={})

    config_use_goodreads = Column(Boolean, default=False)
    config_goodreads_api_key = Column(String)
    config_goodreads_api_secret = Column(String)
-
+    config_register_email = Column(Boolean, default=False)
    config_login_type = Column(Integer, default=0)

    config_kobo_proxy = Column(Boolean, default=False)

-
    config_ldap_provider_url = Column(String, default='example.org')
    config_ldap_port = Column(SmallInteger, default=389)
    config_ldap_authentication = Column(SmallInteger, default=constants.LDAP_AUTH_SIMPLE)
@@ -116,10 +117,11 @@ class _Settings(_Base):
    config_ldap_group_members_field = Column(String, default='memberUid')
    config_ldap_group_name = Column(String, default='calibreweb')

-    config_ebookconverter = Column(Integer, default=0)
-    config_converterpath = Column(String)
+    config_kepubifypath = Column(String, default=None)
+    config_converterpath = Column(String, default=None)
    config_calibre = Column(String)
-    config_rarfile_location = Column(String)
+    config_rarfile_location = Column(String, default=None)
+    config_upload_formats = Column(String, default=','.join(constants.EXTENSIONS_UPLOAD))

    config_updatechannel = Column(Integer, default=constants.UPDATE_STABLE)

@@ -140,6 +142,22 @@ class _ConfigSQL(object):
        self.config_calibre_dir = None
        self.load()

+        change = False
+        if self.config_converterpath == None:
+            change = True
+            self.config_converterpath = autodetect_calibre_binary()
+
+        if self.config_kepubifypath == None:
+            change = True
+            self.config_kepubifypath = autodetect_kepubify_binary()
+
+        if self.config_rarfile_location == None:
+            change = True
+            self.config_rarfile_location = autodetect_unrar_binary()
+        if change:
+            self.save()
+
+
    def _read_from_storage(self):
        if self._settings is None:
            log.debug("_ConfigSQL._read_from_storage")
@@ -197,20 +215,20 @@ class _ConfigSQL(object):
        return self.show_element_new_user(constants.DETAIL_RANDOM)

    def list_denied_tags(self):
-        mct = self.config_denied_tags.split(",")
-        return [t.strip() for t in mct]
+        mct = self.config_denied_tags or ""
+        return [t.strip() for t in mct.split(",")]

    def list_allowed_tags(self):
-        mct = self.config_allowed_tags.split(",")
-        return [t.strip() for t in mct]
+        mct = self.config_allowed_tags or ""
+        return [t.strip() for t in mct.split(",")]

    def list_denied_column_values(self):
-        mct = self.config_denied_column_value.split(",")
-        return [t.strip() for t in mct]
+        mct = self.config_denied_column_value or ""
+        return [t.strip() for t in mct.split(",")]

    def list_allowed_column_values(self):
-        mct = self.config_allowed_column_value.split(",")
-        return [t.strip() for t in mct]
+        mct = self.config_allowed_column_value or ""
+        return [t.strip() for t in mct.split(",")]

    def get_log_level(self):
        return logger.get_level_name(self.config_log_level)
@@ -263,17 +281,19 @@ class _ConfigSQL(object):
                        v = column.default.arg
                setattr(self, k, v)

-        if self.config_google_drive_watch_changes_response:
-            self.config_google_drive_watch_changes_response = json.loads(self.config_google_drive_watch_changes_response)
-
        have_metadata_db = bool(self.config_calibre_dir)
        if have_metadata_db:
            if not self.config_use_google_drive:
                db_file = os.path.join(self.config_calibre_dir, 'metadata.db')
                have_metadata_db = os.path.isfile(db_file)
        self.db_configured = have_metadata_db
-
-        logger.setup(self.config_logfile, self.config_log_level)
+        constants.EXTENSIONS_UPLOAD = [x.lstrip().rstrip() for x in self.config_upload_formats.split(',')]
+        logfile = logger.setup(self.config_logfile, self.config_log_level)
+        if logfile != self.config_logfile:
+            log.warning("Log path %s not valid, falling back to default", self.config_logfile)
+            self.config_logfile = logfile
+            self._session.merge(s)
+            self._session.commit()

    def save(self):
        '''Apply all configuration values to the underlying storage.'''
@@ -330,17 +350,41 @@ def _migrate_table(session, orm_class):
    if changed:
        session.commit()

+
 def autodetect_calibre_binary():
    if sys.platform == "win32":
-        calibre_path = ["C:\\program files\calibre\calibre-convert.exe",
-                        "C:\\program files(x86)\calibre\calibre-convert.exe"]
+        calibre_path = ["C:\\program files\\calibre\\ebook-convert.exe",
+                        "C:\\program files(x86)\\calibre\\ebook-convert.exe",
+                        "C:\\program files(x86)\\calibre2\\ebook-convert.exe",
+                        "C:\\program files\\calibre2\\ebook-convert.exe"]
    else:
        calibre_path = ["/opt/calibre/ebook-convert"]
    for element in calibre_path:
        if os.path.isfile(element) and os.access(element, os.X_OK):
            return element
-    return None
+    return ""

+def autodetect_unrar_binary():
+    if sys.platform == "win32":
+        calibre_path = ["C:\\program files\\WinRar\\unRAR.exe",
+                        "C:\\program files(x86)\\WinRar\\unRAR.exe"]
+    else:
+        calibre_path = ["/usr/bin/unrar"]
+    for element in calibre_path:
+        if os.path.isfile(element) and os.access(element, os.X_OK):
+            return element
+    return ""
+
+def autodetect_kepubify_binary():
+    if sys.platform == "win32":
+        calibre_path = ["C:\\program files\\kepubify\\kepubify-windows-64Bit.exe",
+                        "C:\\program files(x86)\\kepubify\\kepubify-windows-64Bit.exe"]
+    else:
+        calibre_path = ["/opt/kepubify/kepubify-linux-64bit", "/opt/kepubify/kepubify-linux-32bit"]
+    for element in calibre_path:
+        if os.path.isfile(element) and os.access(element, os.X_OK):
+            return element
+    return ""

 def _migrate_database(session):
    # make sure the table is created, if it does not exist
@@ -81,6 +81,7 @@ SIDEBAR_PUBLISHER       = 1 << 12
 SIDEBAR_RATING          = 1 << 13
 SIDEBAR_FORMAT          = 1 << 14
 SIDEBAR_ARCHIVED        = 1 << 15
+# SIDEBAR_LIST            = 1 << 16

 ADMIN_USER_ROLES        = sum(r for r in ALL_ROLES.values()) & ~ROLE_ANONYMOUS
 ADMIN_USER_SIDEBAR      = (SIDEBAR_ARCHIVED << 1) - 1
@@ -110,12 +111,10 @@ except ValueError:
 del env_CALIBRE_PORT


-EXTENSIONS_AUDIO    = {'mp3', 'm4a', 'm4b'}
-EXTENSIONS_CONVERT  = {'pdf', 'epub', 'mobi', 'azw3', 'docx', 'rtf', 'fb2', 'lit', 'lrf', 'txt', 'htmlz', 'rtf', 'odt'}
-EXTENSIONS_UPLOAD   = {'txt', 'pdf', 'epub', 'mobi', 'azw', 'azw3', 'cbr', 'cbz', 'cbt', 'djvu', 'prc', 'doc', 'docx',
-                       'fb2', 'html', 'rtf', 'lit', 'odt', 'mp3',  'm4a', 'm4b'}
-# EXTENSIONS_READER = set(['txt', 'pdf', 'epub', 'zip', 'cbz', 'tar', 'cbt'] +
-#                         (['rar','cbr'] if feature_support['rar'] else []))
+EXTENSIONS_AUDIO    = {'mp3', 'mp4', 'ogg', 'opus', 'wav', 'flac', 'm4a', 'm4b'}
+EXTENSIONS_CONVERT  = ['pdf', 'epub', 'mobi', 'azw3', 'docx', 'rtf', 'fb2', 'lit', 'lrf', 'txt', 'htmlz', 'rtf', 'odt']
+EXTENSIONS_UPLOAD   = {'txt', 'pdf', 'epub', 'kepub', 'mobi', 'azw', 'azw3', 'cbr', 'cbz', 'cbt', 'djvu', 'prc', 'doc', 'docx',
+                       'fb2', 'html', 'rtf', 'lit', 'odt', 'mp3', 'mp4', 'ogg', 'opus', 'wav', 'flac', 'm4a', 'm4b'}


 def has_flag(value, bit_flag):
@@ -129,7 +128,7 @@ def selected_roles(dictionary):
 BookMeta = namedtuple('BookMeta', 'file_path, extension, title, author, cover, description, tags, series, '
                                  'series_id, languages')

-STABLE_VERSION = {'version': '0.6.7'}
+STABLE_VERSION = {'version': '0.6.9'}

 NIGHTLY_VERSION = {}
 NIGHTLY_VERSION[0] = '$Format:%H$'
@@ -19,6 +19,7 @@
 from __future__ import division, print_function, unicode_literals
 import os
 import re
+import sys
 from flask_babel import gettext as _

 from . import config, logger
@@ -29,8 +30,8 @@ log = logger.create()

 # _() necessary to make babel aware of string for translation
 _NOT_CONFIGURED = _('not configured')
-_NOT_INSTALLED = 'not installed'
-_EXECUTION_ERROR = 'Execution permissions missing'
+_NOT_INSTALLED = _('not installed')
+_EXECUTION_ERROR = _('Execution permissions missing')


 def _get_command_version(path, pattern, argument=None):
@@ -48,10 +49,15 @@ def _get_command_version(path, pattern, argument=None):
    return _NOT_INSTALLED


-def get_version():
-    version = None
-    if config.config_ebookconverter == 1:
-        version = _get_command_version(config.config_converterpath, r'Amazon kindlegen\(')
-    elif config.config_ebookconverter == 2:
-        version = _get_command_version(config.config_converterpath, r'ebook-convert.*\(calibre', '--version')
-    return version or _NOT_CONFIGURED
+def get_calibre_version():
+    return _get_command_version(config.config_converterpath, r'ebook-convert.*\(calibre', '--version') \
+           or _NOT_CONFIGURED
+
+
+def get_unrar_version():
+    return _get_command_version(config.config_rarfile_location, r'UNRAR.*\d') or _NOT_CONFIGURED
+
+def get_kepubify_version():
+    return _get_command_version(config.config_kepubifypath, r'kepubify\s','--version') or _NOT_CONFIGURED
+
+
@@ -22,18 +22,36 @@ import sys
 import os
 import re
 import ast
+import json
+from datetime import datetime
+import threading

 from sqlalchemy import create_engine
-from sqlalchemy import Table, Column, ForeignKey
-from sqlalchemy import String, Integer, Boolean, TIMESTAMP, Float, DateTime
+from sqlalchemy import Table, Column, ForeignKey, CheckConstraint
+from sqlalchemy import String, Integer, Boolean, TIMESTAMP, Float
 from sqlalchemy.orm import relationship, sessionmaker, scoped_session
 from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.exc import OperationalError
+from sqlalchemy.pool import StaticPool
+from flask_login import current_user
+from sqlalchemy.sql.expression import and_, true, false, text, func, or_
+from sqlalchemy.ext.associationproxy import association_proxy
+from babel import Locale as LC
+from babel.core import UnknownLocaleError
+from flask_babel import gettext as _
+
+from . import logger, ub, isoLanguages
+from .pagination import Pagination
+
+try:
+    import unidecode
+    use_unidecode = True
+except ImportError:
+    use_unidecode = False


-session = None
 cc_exceptions = ['datetime', 'comments', 'composite', 'series']
 cc_classes = {}
-engine = None

 Base = declarative_base()

@@ -72,9 +90,9 @@ class Identifiers(Base):
    __tablename__ = 'identifiers'

    id = Column(Integer, primary_key=True)
-    type = Column(String)
-    val = Column(String)
-    book = Column(Integer, ForeignKey('books.id'))
+    type = Column(String(collation='NOCASE'), nullable=False, default="isbn")
+    val = Column(String(collation='NOCASE'), nullable=False)
+    book = Column(Integer, ForeignKey('books.id'), nullable=False)

    def __init__(self, val, id_type, book):
        self.val = val
@@ -82,41 +100,61 @@ class Identifiers(Base):
        self.book = book

    def formatType(self):
-        if self.type == "amazon":
+        format_type = self.type.lower()
+        if format_type == 'amazon':
            return u"Amazon"
-        elif self.type == "isbn":
+        elif format_type.startswith("amazon_"):
+            return u"Amazon.{0}".format(format_type[7:])
+        elif format_type == "isbn":
            return u"ISBN"
-        elif self.type == "doi":
+        elif format_type == "doi":
            return u"DOI"
-        elif self.type == "goodreads":
+        elif format_type == "douban":
+            return u"Douban"
+        elif format_type == "goodreads":
            return u"Goodreads"
-        elif self.type == "google":
+        elif format_type == "google":
            return u"Google Books"
-        elif self.type == "kobo":
+        elif format_type == "kobo":
            return u"Kobo"
-        if self.type == "lubimyczytac":
+        elif format_type == "litres":
+            return u"ЛитРес"
+        elif format_type == "issn":
+            return u"ISSN"
+        elif format_type == "isfdb":
+            return u"ISFDB"
+        if format_type == "lubimyczytac":
            return u"Lubimyczytac"
        else:
            return self.type

    def __repr__(self):
-        if self.type == "amazon":
-            return u"https://amzn.com/{0}".format(self.val)
-        elif self.type == "isbn":
-            return u"http://www.worldcat.org/isbn/{0}".format(self.val)
-        elif self.type == "doi":
-            return u"http://dx.doi.org/{0}".format(self.val)
-        elif self.type == "goodreads":
-            return u"http://www.goodreads.com/book/show/{0}".format(self.val)
-        elif self.type == "douban":
+        format_type = self.type.lower()
+        if format_type == "amazon" or format_type == "asin":
+            return u"https://amazon.com/dp/{0}".format(self.val)
+        elif format_type.startswith('amazon_'):
+            return u"https://amazon.{0}/dp/{1}".format(format_type[7:], self.val)
+        elif format_type == "isbn":
+            return u"https://www.worldcat.org/isbn/{0}".format(self.val)
+        elif format_type == "doi":
+            return u"https://dx.doi.org/{0}".format(self.val)
+        elif format_type == "goodreads":
+            return u"https://www.goodreads.com/book/show/{0}".format(self.val)
+        elif format_type == "douban":
            return u"https://book.douban.com/subject/{0}".format(self.val)
-        elif self.type == "google":
+        elif format_type == "google":
            return u"https://books.google.com/books?id={0}".format(self.val)
-        elif self.type == "kobo":
+        elif format_type == "kobo":
            return u"https://www.kobo.com/ebook/{0}".format(self.val)
-        elif self.type == "lubimyczytac":
-            return u" http://lubimyczytac.pl/ksiazka/{0}".format(self.val)
-        elif self.type == "url":
+        elif format_type == "lubimyczytac":
+            return u" https://lubimyczytac.pl/ksiazka/{0}/ksiazka".format(self.val)
+        elif format_type == "litres":
+            return u"https://www.litres.ru/{0}".format(self.val)
+        elif format_type == "issn":
+            return u"https://portal.issn.org/resource/ISSN/{0}".format(self.val)
+        elif format_type == "isfdb":
+            return u"http://www.isfdb.org/cgi-bin/pl.cgi?{0}".format(self.val)
+        elif format_type == "url":
            return u"{0}".format(self.val)
        else:
            return u""
@@ -126,8 +164,8 @@ class Comments(Base):
    __tablename__ = 'comments'

    id = Column(Integer, primary_key=True)
-    text = Column(String)
-    book = Column(Integer, ForeignKey('books.id'))
+    text = Column(String(collation='NOCASE'), nullable=False)
+    book = Column(Integer, ForeignKey('books.id'), nullable=False)

    def __init__(self, text, book):
        self.text = text
@@ -141,7 +179,7 @@ class Tags(Base):
    __tablename__ = 'tags'

    id = Column(Integer, primary_key=True, autoincrement=True)
-    name = Column(String)
+    name = Column(String(collation='NOCASE'), unique=True, nullable=False)

    def __init__(self, name):
        self.name = name
@@ -154,9 +192,9 @@ class Authors(Base):
    __tablename__ = 'authors'

    id = Column(Integer, primary_key=True)
-    name = Column(String)
-    sort = Column(String)
-    link = Column(String)
+    name = Column(String(collation='NOCASE'), unique=True, nullable=False)
+    sort = Column(String(collation='NOCASE'))
+    link = Column(String, nullable=False, default="")

    def __init__(self, name, sort, link):
        self.name = name
@@ -171,8 +209,8 @@ class Series(Base):
    __tablename__ = 'series'

    id = Column(Integer, primary_key=True)
-    name = Column(String)
-    sort = Column(String)
+    name = Column(String(collation='NOCASE'), unique=True, nullable=False)
+    sort = Column(String(collation='NOCASE'))

    def __init__(self, name, sort):
        self.name = name
@@ -186,7 +224,7 @@ class Ratings(Base):
    __tablename__ = 'ratings'

    id = Column(Integer, primary_key=True)
-    rating = Column(Integer)
+    rating = Column(Integer, CheckConstraint('rating>-1 AND rating<11'), unique=True)

    def __init__(self, rating):
        self.rating = rating
@@ -199,7 +237,7 @@ class Languages(Base):
    __tablename__ = 'languages'

    id = Column(Integer, primary_key=True)
-    lang_code = Column(String)
+    lang_code = Column(String(collation='NOCASE'), nullable=False, unique=True)

    def __init__(self, lang_code):
        self.lang_code = lang_code
@@ -212,8 +250,8 @@ class Publishers(Base):
    __tablename__ = 'publishers'

    id = Column(Integer, primary_key=True)
-    name = Column(String)
-    sort = Column(String)
+    name = Column(String(collation='NOCASE'), nullable=False, unique=True)
+    sort = Column(String(collation='NOCASE'))

    def __init__(self, name, sort):
        self.name = name
@@ -225,12 +263,13 @@ class Publishers(Base):

 class Data(Base):
    __tablename__ = 'data'
+    __table_args__ = {'schema':'calibre'}

    id = Column(Integer, primary_key=True)
-    book = Column(Integer, ForeignKey('books.id'))
-    format = Column(String)
-    uncompressed_size = Column(Integer)
-    name = Column(String)
+    book = Column(Integer, ForeignKey('books.id'), nullable=False)
+    format = Column(String(collation='NOCASE'), nullable=False)
+    uncompressed_size = Column(Integer, nullable=False)
+    name = Column(String, nullable=False)

    def __init__(self, book, book_format, uncompressed_size, name):
        self.book = book
@@ -247,20 +286,23 @@ class Books(Base):

    DEFAULT_PUBDATE = "0101-01-01 00:00:00+00:00"

-    id = Column(Integer, primary_key=True)
-    title = Column(String)
-    sort = Column(String)
-    author_sort = Column(String)
-    timestamp = Column(TIMESTAMP)
-    pubdate = Column(String)
-    series_index = Column(String)
-    last_modified = Column(TIMESTAMP)
-    path = Column(String)
-    has_cover = Column(Integer)
+    id = Column(Integer, primary_key=True, autoincrement=True)
+    title = Column(String(collation='NOCASE'), nullable=False, default='Unknown')
+    sort = Column(String(collation='NOCASE'))
+    author_sort = Column(String(collation='NOCASE'))
+    timestamp = Column(TIMESTAMP, default=datetime.utcnow)
+    pubdate = Column(String) # , default=datetime.utcnow)
+    series_index = Column(String, nullable=False, default="1.0")
+    last_modified = Column(TIMESTAMP, default=datetime.utcnow)
+    path = Column(String, default="", nullable=False)
+    has_cover = Column(Integer, default=0)
    uuid = Column(String)
+    isbn = Column(String(collation='NOCASE'), default="")
+    # Iccn = Column(String(collation='NOCASE'), default="")
+    flags = Column(Integer, nullable=False, default=1)

    authors = relationship('Authors', secondary=books_authors_link, backref='books')
-    tags = relationship('Tags', secondary=books_tags_link, backref='books',order_by="Tags.name")
+    tags = relationship('Tags', secondary=books_tags_link, backref='books', order_by="Tags.name")
    comments = relationship('Comments', backref='books')
    data = relationship('Data', backref='books')
    series = relationship('Series', secondary=books_series_link, backref='books')
@@ -310,131 +352,339 @@ class Custom_Columns(Base):
        return display_dict


-def update_title_sort(config, conn=None):
-    # user defined sort function for calibre databases (Series, etc.)
-    def _title_sort(title):
-        # calibre sort stuff
-        title_pat = re.compile(config.config_title_regex, re.IGNORECASE)
-        match = title_pat.search(title)
-        if match:
-            prep = match.group(1)
-            title = title.replace(prep, '') + ', ' + prep
-        return title.strip()
+class CalibreDB(threading.Thread):

-    conn = conn or session.connection().connection.connection
-    conn.create_function("title_sort", 1, _title_sort)
+    def __init__(self):
+        threading.Thread.__init__(self)
+        self.engine = None
+        self.session = None
+        self.queue = None
+        self.log = None
+        self.config = None
+
+    def add_queue(self,queue):
+        self.queue = queue
+        self.log = logger.create()
+
+    def run(self):
+        while True:
+            i = self.queue.get()
+            if i == 'dummy':
+                self.queue.task_done()
+                break
+            if i['task'] == 'add_format':
+                cur_book = self.session.query(Books).filter(Books.id == i['id']).first()
+                cur_book.data.append(i['format'])
+                try:
+                    # db.session.merge(cur_book)
+                    self.session.commit()
+                except OperationalError as e:
+                    self.session.rollback()
+                    self.log.error("Database error: %s", e)
+                    # self._handleError(_(u"Database error: %(error)s.", error=e))
+                    # return
+            self.queue.task_done()


-def setup_db(config):
-    dispose()
-    global engine
+    def stop(self):
+        self.queue.put('dummy')

-    if not config.config_calibre_dir:
-        config.invalidate()
-        return False
+    def setup_db(self, config, app_db_path):
+        self.config = config
+        self.dispose()

-    dbpath = os.path.join(config.config_calibre_dir, "metadata.db")
-    if not os.path.exists(dbpath):
-        config.invalidate()
-        return False
+        if not config.config_calibre_dir:
+            config.invalidate()
+            return False

-    try:
-        engine = create_engine('sqlite:///{0}'.format(dbpath),
-                               echo=False,
-                               isolation_level="SERIALIZABLE",
-                               connect_args={'check_same_thread': False})
-        conn = engine.connect()
-        # conn.text_factory = lambda b: b.decode(errors = 'ignore') possible fix for #1302
-    except Exception as e:
-        config.invalidate(e)
-        return False
+        dbpath = os.path.join(config.config_calibre_dir, "metadata.db")
+        if not os.path.exists(dbpath):
+            config.invalidate()
+            return False

-    config.db_configured = True
-    update_title_sort(config, conn.connection)
+        try:
+            self.engine = create_engine('sqlite://',
+                                   echo=False,
+                                   isolation_level="SERIALIZABLE",
+                                   connect_args={'check_same_thread': False},
+                                   poolclass=StaticPool)
+            self.engine.execute("attach database '{}' as calibre;".format(dbpath))
+            self.engine.execute("attach database '{}' as app_settings;".format(app_db_path))

-    if not cc_classes:
-        cc = conn.execute("SELECT id, datatype FROM custom_columns")
+            conn = self.engine.connect()
+            # conn.text_factory = lambda b: b.decode(errors = 'ignore') possible fix for #1302
+        except Exception as e:
+            config.invalidate(e)
+            return False
+
+        config.db_configured = True
+        self.update_title_sort(config, conn.connection)
+
+        if not cc_classes:
+            cc = conn.execute("SELECT id, datatype FROM custom_columns")
+
+            cc_ids = []
+            books_custom_column_links = {}
+            for row in cc:
+                if row.datatype not in cc_exceptions:
+                    if row.datatype == 'series':
+                        dicttable = {'__tablename__': 'books_custom_column_' + str(row.id) + '_link',
+                                     'id': Column(Integer, primary_key=True),
+                                     'book': Column(Integer, ForeignKey('books.id'),
+                                                    primary_key=True),
+                                     'map_value': Column('value', Integer,
+                                                     ForeignKey('custom_column_' +
+                                                                str(row.id) + '.id'),
+                                                     primary_key=True),
+                                     'extra': Column(Float),
+                                     'asoc' : relationship('custom_column_' + str(row.id), uselist=False),
+                                     'value' : association_proxy('asoc', 'value')
+                                     }
+                        books_custom_column_links[row.id] = type(str('books_custom_column_' + str(row.id) + '_link'),
+                                                                 (Base,), dicttable)
+                    else:
+                        books_custom_column_links[row.id] = Table('books_custom_column_' + str(row.id) + '_link',
+                                                                  Base.metadata,
+                                                                  Column('book', Integer, ForeignKey('books.id'),
+                                                                         primary_key=True),
+                                                                  Column('value', Integer,
+                                                                         ForeignKey('custom_column_' +
+                                                                                    str(row.id) + '.id'),
+                                                                         primary_key=True)
+                                                                  )
+                    cc_ids.append([row.id, row.datatype])

-        cc_ids = []
-        books_custom_column_links = {}
-        for row in cc:
-            if row.datatype not in cc_exceptions:
-                books_custom_column_links[row.id] = Table('books_custom_column_' + str(row.id) + '_link', Base.metadata,
-                                                          Column('book', Integer, ForeignKey('books.id'),
-                                                                 primary_key=True),
-                                                          Column('value', Integer,
-                                                                 ForeignKey('custom_column_' + str(row.id) + '.id'),
-                                                                 primary_key=True)
-                                                          )
-                cc_ids.append([row.id, row.datatype])
-                if row.datatype == 'bool':
                    ccdict = {'__tablename__': 'custom_column_' + str(row.id),
-                              'id': Column(Integer, primary_key=True),
-                              'book': Column(Integer, ForeignKey('books.id')),
-                              'value': Column(Boolean)}
-                elif row.datatype == 'int':
-                    ccdict = {'__tablename__': 'custom_column_' + str(row.id),
-                              'id': Column(Integer, primary_key=True),
-                              'book': Column(Integer, ForeignKey('books.id')),
-                              'value': Column(Integer)}
-                elif row.datatype == 'float':
-                    ccdict = {'__tablename__': 'custom_column_' + str(row.id),
-                              'id': Column(Integer, primary_key=True),
-                              'book': Column(Integer, ForeignKey('books.id')),
-                              'value': Column(Float)}
+                              'id': Column(Integer, primary_key=True)}
+                    if row.datatype == 'float':
+                        ccdict['value'] = Column(Float)
+                    elif row.datatype == 'int':
+                        ccdict['value'] = Column(Integer)
+                    elif row.datatype == 'bool':
+                        ccdict['value'] = Column(Boolean)
+                    else:
+                        ccdict['value'] = Column(String)
+                    if row.datatype in ['float', 'int', 'bool']:
+                        ccdict['book'] = Column(Integer, ForeignKey('books.id'))
+                    cc_classes[row.id] = type(str('custom_column_' + str(row.id)), (Base,), ccdict)
+
+            for cc_id in cc_ids:
+                if (cc_id[1] == 'bool') or (cc_id[1] == 'int') or (cc_id[1] == 'float'):
+                    setattr(Books,
+                            'custom_column_' + str(cc_id[0]),
+                            relationship(cc_classes[cc_id[0]],
+                                         primaryjoin=(
+                                         Books.id == cc_classes[cc_id[0]].book),
+                                         backref='books'))
+                elif (cc_id[1] == 'series'):
+                    setattr(Books,
+                            'custom_column_' + str(cc_id[0]),
+                            relationship(books_custom_column_links[cc_id[0]],
+                                         backref='books'))
                else:
-                    ccdict = {'__tablename__': 'custom_column_' + str(row.id),
-                              'id': Column(Integer, primary_key=True),
-                              'value': Column(String)}
-                cc_classes[row.id] = type(str('Custom_Column_' + str(row.id)), (Base,), ccdict)
+                    setattr(Books,
+                            'custom_column_' + str(cc_id[0]),
+                            relationship(cc_classes[cc_id[0]],
+                                         secondary=books_custom_column_links[cc_id[0]],
+                                         backref='books'))

-        for cc_id in cc_ids:
-            if (cc_id[1] == 'bool') or (cc_id[1] == 'int') or (cc_id[1] == 'float'):
-                setattr(Books, 'custom_column_' + str(cc_id[0]), relationship(cc_classes[cc_id[0]],
-                                                                           primaryjoin=(
-                                                                           Books.id == cc_classes[cc_id[0]].book),
-                                                                           backref='books'))
-            else:
-                setattr(Books, 'custom_column_' + str(cc_id[0]), relationship(cc_classes[cc_id[0]],
-                                                                           secondary=books_custom_column_links[cc_id[0]],
-                                                                           backref='books'))
+        Session = scoped_session(sessionmaker(autocommit=False,
+                                              autoflush=False,
+                                              bind=self.engine))
+        self.session = Session()
+        return True

+    def get_book(self, book_id):
+        return self.session.query(Books).filter(Books.id == book_id).first()

-    global session
-    Session = scoped_session(sessionmaker(autocommit=False,
-                                             autoflush=False,
-                                             bind=engine))
-    session = Session()
-    return True
+    def get_filtered_book(self, book_id, allow_show_archived=False):
+        return self.session.query(Books).filter(Books.id == book_id).\
+            filter(self.common_filters(allow_show_archived)).first()

+    def get_book_by_uuid(self, book_uuid):
+        return self.session.query(Books).filter(Books.uuid == book_uuid).first()

-def dispose():
-    global session
+    def get_book_format(self, book_id, format):
+        return self.session.query(Data).filter(Data.book == book_id).filter(Data.format == format).first()

-    old_session = session
-    session = None
-    if old_session:
-        try: old_session.close()
-        except: pass
-        if old_session.bind:
-            try: old_session.bind.dispose()
-            except Exception: pass
+    # Language and content filters for displaying in the UI
+    def common_filters(self, allow_show_archived=False):
+        if not allow_show_archived:
+            archived_books = (
+                ub.session.query(ub.ArchivedBook)
+                    .filter(ub.ArchivedBook.user_id == int(current_user.id))
+                    .filter(ub.ArchivedBook.is_archived == True)
+                    .all()
+            )
+            archived_book_ids = [archived_book.book_id for archived_book in archived_books]
+            archived_filter = Books.id.notin_(archived_book_ids)
+        else:
+            archived_filter = true()

-    for attr in list(Books.__dict__.keys()):
-        if attr.startswith("custom_column_"):
-            setattr(Books, attr, None)
+        if current_user.filter_language() != "all":
+            lang_filter = Books.languages.any(Languages.lang_code == current_user.filter_language())
+        else:
+            lang_filter = true()
+        negtags_list = current_user.list_denied_tags()
+        postags_list = current_user.list_allowed_tags()
+        neg_content_tags_filter = false() if negtags_list == [''] else Books.tags.any(Tags.name.in_(negtags_list))
+        pos_content_tags_filter = true() if postags_list == [''] else Books.tags.any(Tags.name.in_(postags_list))
+        if self.config.config_restricted_column:
+            pos_cc_list = current_user.allowed_column_value.split(',')
+            pos_content_cc_filter = true() if pos_cc_list == [''] else \
+                getattr(Books, 'custom_column_' + str(self.config.config_restricted_column)). \
+                    any(cc_classes[self.config.config_restricted_column].value.in_(pos_cc_list))
+            neg_cc_list = current_user.denied_column_value.split(',')
+            neg_content_cc_filter = false() if neg_cc_list == [''] else \
+                getattr(Books, 'custom_column_' + str(self.config.config_restricted_column)). \
+                    any(cc_classes[self.config.config_restricted_column].value.in_(neg_cc_list))
+        else:
+            pos_content_cc_filter = true()
+            neg_content_cc_filter = false()
+        return and_(lang_filter, pos_content_tags_filter, ~neg_content_tags_filter,
+                    pos_content_cc_filter, ~neg_content_cc_filter, archived_filter)

-    for db_class in cc_classes.values():
-        Base.metadata.remove(db_class.__table__)
-    cc_classes.clear()
+    # Fill indexpage with all requested data from database
+    def fill_indexpage(self, page, database, db_filter, order, *join):
+        return self.fill_indexpage_with_archived_books(page, database, db_filter, order, False, *join)

-    for table in reversed(Base.metadata.sorted_tables):
-        name = table.key
-        if name.startswith("custom_column_") or name.startswith("books_custom_column_"):
-            if table is not None:
-                Base.metadata.remove(table)
+    def fill_indexpage_with_archived_books(self, page, database, db_filter, order, allow_show_archived, *join):
+        if current_user.show_detail_random():
+            randm = self.session.query(Books) \
+                .filter(self.common_filters(allow_show_archived)) \
+                .order_by(func.random()) \
+                .limit(self.config.config_random_books)
+        else:
+            randm = false()
+        off = int(int(self.config.config_books_per_page) * (page - 1))
+        query = self.session.query(database) \
+            .join(*join, isouter=True) \
+            .filter(db_filter) \
+            .filter(self.common_filters(allow_show_archived))
+        pagination = Pagination(page, self.config.config_books_per_page,
+                                len(query.all()))
+        entries = query.order_by(*order).offset(off).limit(self.config.config_books_per_page).all()
+        for book in entries:
+            book = self.order_authors(book)
+        return entries, randm, pagination

-def reconnect_db(config):
-    session.close()
-    engine.dispose()
-    setup_db(config)
+    # Orders all Authors in the list according to authors sort
+    def order_authors(self, entry):
+        sort_authors = entry.author_sort.split('&')
+        authors_ordered = list()
+        error = False
+        for auth in sort_authors:
+            # ToDo: How to handle not found authorname
+            result = self.session.query(Authors).filter(Authors.sort == auth.lstrip().strip()).first()
+            if not result:
+                error = True
+                break
+            authors_ordered.append(result)
+        if not error:
+            entry.authors = authors_ordered
+        return entry
+
+    def get_typeahead(self, database, query, replace=('', ''), tag_filter=true()):
+        query = query or ''
+        self.session.connection().connection.connection.create_function("lower", 1, lcase)
+        entries = self.session.query(database).filter(tag_filter). \
+            filter(func.lower(database.name).ilike("%" + query + "%")).all()
+        json_dumps = json.dumps([dict(name=r.name.replace(*replace)) for r in entries])
+        return json_dumps
+
+    def check_exists_book(self, authr, title):
+        self.session.connection().connection.connection.create_function("lower", 1, lcase)
+        q = list()
+        authorterms = re.split(r'\s*&\s*', authr)
+        for authorterm in authorterms:
+            q.append(Books.authors.any(func.lower(Authors.name).ilike("%" + authorterm + "%")))
+
+        return self.session.query(Books)\
+            .filter(and_(Books.authors.any(and_(*q)), func.lower(Books.title).ilike("%" + title + "%"))).first()
+
+    # read search results from calibre-database and return it (function is used for feed and simple search
+    def get_search_results(self, term):
+        term.strip().lower()
+        self.session.connection().connection.connection.create_function("lower", 1, lcase)
+        q = list()
+        authorterms = re.split("[, ]+", term)
+        for authorterm in authorterms:
+            q.append(Books.authors.any(func.lower(Authors.name).ilike("%" + authorterm + "%")))
+        return self.session.query(Books).filter(self.common_filters(True)).filter(
+            or_(Books.tags.any(func.lower(Tags.name).ilike("%" + term + "%")),
+                Books.series.any(func.lower(Series.name).ilike("%" + term + "%")),
+                Books.authors.any(and_(*q)),
+                Books.publishers.any(func.lower(Publishers.name).ilike("%" + term + "%")),
+                func.lower(Books.title).ilike("%" + term + "%")
+                )).order_by(Books.sort).all()
+
+    # Creates for all stored languages a translated speaking name in the array for the UI
+    def speaking_language(self, languages=None):
+        from . import get_locale
+
+        if not languages:
+            languages = self.session.query(Languages) \
+                .join(books_languages_link) \
+                .join(Books) \
+                .filter(self.common_filters()) \
+                .group_by(text('books_languages_link.lang_code')).all()
+        for lang in languages:
+            try:
+                cur_l = LC.parse(lang.lang_code)
+                lang.name = cur_l.get_language_name(get_locale())
+            except UnknownLocaleError:
+                lang.name = _(isoLanguages.get(part3=lang.lang_code).name)
+        return languages
+
+    def update_title_sort(self, config, conn=None):
+        # user defined sort function for calibre databases (Series, etc.)
+        def _title_sort(title):
+            # calibre sort stuff
+            title_pat = re.compile(config.config_title_regex, re.IGNORECASE)
+            match = title_pat.search(title)
+            if match:
+                prep = match.group(1)
+                title = title[len(prep):] + ', ' + prep
+            return title.strip()
+
+        conn = conn or self.session.connection().connection.connection
+        conn.create_function("title_sort", 1, _title_sort)
+
+    def dispose(self):
+        # global session
+
+        old_session = self.session
+        self.session = None
+        if old_session:
+            try: old_session.close()
+            except: pass
+            if old_session.bind:
+                try: old_session.bind.dispose()
+                except Exception: pass
+
+        for attr in list(Books.__dict__.keys()):
+            if attr.startswith("custom_column_"):
+                setattr(Books, attr, None)
+
+        for db_class in cc_classes.values():
+            Base.metadata.remove(db_class.__table__)
+        cc_classes.clear()
+
+        for table in reversed(Base.metadata.sorted_tables):
+            name = table.key
+            if name.startswith("custom_column_") or name.startswith("books_custom_column_"):
+                if table is not None:
+                    Base.metadata.remove(table)
+
+    def reconnect_db(self, config, app_db_path):
+        self.session.close()
+        self.engine.dispose()
+        self.setup_db(config, app_db_path)
+
+def lcase(s):
+    try:
+        return unidecode.unidecode(s.lower())
+    except Exception as e:
+        log = logger.create()
+        log.exception(e)
+        return s.lower()
@@ -24,16 +24,17 @@ from __future__ import division, print_function, unicode_literals
 import os
 from datetime import datetime
 import json
-from shutil import move, copyfile
+from shutil import copyfile
 from uuid import uuid4

 from flask import Blueprint, request, flash, redirect, url_for, abort, Markup, Response
 from flask_babel import gettext as _
 from flask_login import current_user, login_required
+from sqlalchemy.exc import OperationalError

 from . import constants, logger, isoLanguages, gdriveutils, uploader, helper
-from . import config, get_locale, db, ub, worker
-from .helper import order_authors, common_filters
+from . import config, get_locale, ub, worker, db
+from . import calibre_db
 from .web import login_required_if_no_ano, render_title_template, edit_required, upload_required


@@ -150,8 +151,11 @@ def modify_identifiers(input_identifiers, db_identifiers, db_session):
       input_identifiers is a list of read-to-persist Identifiers objects.
       db_identifiers is a list of already persisted list of Identifiers objects."""
    changed = False
-    input_dict = dict([ (identifier.type.lower(), identifier) for identifier in input_identifiers ])
-    db_dict = dict([ (identifier.type.lower(), identifier) for identifier in db_identifiers ])
+    error = False
+    input_dict = dict([(identifier.type.lower(), identifier) for identifier in input_identifiers])
+    if len(input_identifiers) != len(input_dict):
+        error = True
+    db_dict = dict([(identifier.type.lower(), identifier) for identifier in db_identifiers ])
    # delete db identifiers not present in input or modify them with input val
    for identifier_type, identifier in db_dict.items():
        if identifier_type not in input_dict.keys():
@@ -166,7 +170,7 @@ def modify_identifiers(input_identifiers, db_identifiers, db_session):
        if identifier_type not in db_dict.keys():
            db_session.add(identifier)
            changed = True
-    return changed
+    return changed, error


@editbook.route("/delete/<int:book_id>/", defaults={'book_format': ""})
@@ -174,13 +178,15 @@ def modify_identifiers(input_identifiers, db_identifiers, db_session):
@login_required
 def delete_book(book_id, book_format):
    if current_user.role_delete_books():
-        book = db.session.query(db.Books).filter(db.Books.id == book_id).first()
+        book = calibre_db.get_book(book_id)
        if book:
            try:
                result, error = helper.delete_book(book, config.config_calibre_dir, book_format=book_format.upper())
                if not result:
                    flash(error, category="error")
                    return redirect(url_for('editbook.edit_book', book_id=book_id))
+                if error:
+                    flash(error, category="warning")
                if not book_format:
                    # delete book from Shelfs, Downloads, Read list
                    ub.session.query(ub.BookShelf).filter(ub.BookShelf.book_id == book_id).delete()
@@ -190,13 +196,13 @@ def delete_book(book_id, book_format):

                    # check if only this book links to:
                    # author, language, series, tags, custom columns
-                    modify_database_object([u''], book.authors, db.Authors, db.session, 'author')
-                    modify_database_object([u''], book.tags, db.Tags, db.session, 'tags')
-                    modify_database_object([u''], book.series, db.Series, db.session, 'series')
-                    modify_database_object([u''], book.languages, db.Languages, db.session, 'languages')
-                    modify_database_object([u''], book.publishers, db.Publishers, db.session, 'publishers')
+                    modify_database_object([u''], book.authors, db.Authors, calibre_db.session, 'author')
+                    modify_database_object([u''], book.tags, db.Tags, calibre_db.session, 'tags')
+                    modify_database_object([u''], book.series, db.Series, calibre_db.session, 'series')
+                    modify_database_object([u''], book.languages, db.Languages, calibre_db.session, 'languages')
+                    modify_database_object([u''], book.publishers, db.Publishers, calibre_db.session, 'publishers')

-                    cc = db.session.query(db.Custom_Columns).\
+                    cc = calibre_db.session.query(db.Custom_Columns).\
                        filter(db.Custom_Columns.datatype.notin_(db.cc_exceptions)).all()
                    for c in cc:
                        cc_string = "custom_column_" + str(c.id)
@@ -206,32 +212,32 @@ def delete_book(book_id, book_format):
                                    del_cc = getattr(book, cc_string)[0]
                                    getattr(book, cc_string).remove(del_cc)
                                    log.debug('remove ' + str(c.id))
-                                    db.session.delete(del_cc)
-                                    db.session.commit()
+                                    calibre_db.session.delete(del_cc)
+                                    calibre_db.session.commit()
                                elif c.datatype == 'rating':
                                    del_cc = getattr(book, cc_string)[0]
                                    getattr(book, cc_string).remove(del_cc)
                                    if len(del_cc.books) == 0:
                                        log.debug('remove ' + str(c.id))
-                                        db.session.delete(del_cc)
-                                        db.session.commit()
+                                        calibre_db.session.delete(del_cc)
+                                        calibre_db.session.commit()
                                else:
                                    del_cc = getattr(book, cc_string)[0]
                                    getattr(book, cc_string).remove(del_cc)
                                    log.debug('remove ' + str(c.id))
-                                    db.session.delete(del_cc)
-                                    db.session.commit()
+                                    calibre_db.session.delete(del_cc)
+                                    calibre_db.session.commit()
                        else:
                            modify_database_object([u''], getattr(book, cc_string), db.cc_classes[c.id],
-                                                   db.session, 'custom')
-                    db.session.query(db.Books).filter(db.Books.id == book_id).delete()
+                                                   calibre_db.session, 'custom')
+                    calibre_db.session.query(db.Books).filter(db.Books.id == book_id).delete()
                else:
-                    db.session.query(db.Data).filter(db.Data.book == book.id).\
+                    calibre_db.session.query(db.Data).filter(db.Data.book == book.id).\
                        filter(db.Data.format == book_format).delete()
-                db.session.commit()
+                calibre_db.session.commit()
            except Exception as e:
                log.debug(e)
-                db.session.rollback()
+                calibre_db.session.rollback()
        else:
            # book not found
            log.error('Book with id "%s" could not be deleted: not found', book_id)
@@ -244,11 +250,9 @@ def delete_book(book_id, book_format):


 def render_edit_book(book_id):
-    db.update_title_sort(config)
-    cc = db.session.query(db.Custom_Columns).filter(db.Custom_Columns.datatype.notin_(db.cc_exceptions)).all()
-    book = db.session.query(db.Books)\
-        .filter(db.Books.id == book_id).filter(common_filters()).first()
-
+    calibre_db.update_title_sort(config)
+    cc = calibre_db.session.query(db.Custom_Columns).filter(db.Custom_Columns.datatype.notin_(db.cc_exceptions)).all()
+    book = calibre_db.get_filtered_book(book_id)
    if not book:
        flash(_(u"Error opening eBook. File does not exist or file is not accessible"), category="error")
        return redirect(url_for("web.index"))
@@ -256,7 +260,7 @@ def render_edit_book(book_id):
    for lang in book.languages:
        lang.language_name = isoLanguages.get_language_name(get_locale(), lang.lang_code)

-    book = order_authors(book)
+    book = calibre_db.order_authors(book)

    author_names = []
    for authr in book.authors:
@@ -264,19 +268,25 @@ def render_edit_book(book_id):

    # Option for showing convertbook button
    valid_source_formats=list()
-    if config.config_ebookconverter == 2:
+    allowed_conversion_formats = list()
+    kepub_possible=None
+    if config.config_converterpath:
        for file in book.data:
            if file.format.lower() in constants.EXTENSIONS_CONVERT:
                valid_source_formats.append(file.format.lower())
+    if config.config_kepubifypath and 'epub' in [file.format.lower() for file in book.data]:
+        kepub_possible = True
+        if not config.config_converterpath:
+            valid_source_formats.append('epub')

    # Determine what formats don't already exist
-    allowed_conversion_formats = constants.EXTENSIONS_CONVERT.copy()
-    for file in book.data:
-        try:
-            allowed_conversion_formats.remove(file.format.lower())
-        except Exception:
-            log.warning('%s already removed from list.', file.format.lower())
-
+    if config.config_converterpath:
+        allowed_conversion_formats = constants.EXTENSIONS_CONVERT[:]
+        for file in book.data:
+            if file.format.lower() in allowed_conversion_formats:
+                allowed_conversion_formats.remove(file.format.lower())
+    if kepub_possible:
+        allowed_conversion_formats.append('kepub')
    return render_title_template('book_edit.html', book=book, authors=author_names, cc=cc,
                                 title=_(u"edit metadata"), page="editbook",
                                 conversion_formats=allowed_conversion_formats,
@@ -293,7 +303,7 @@ def edit_book_ratings(to_save, book):
        ratingx2 = int(float(to_save["rating"]) * 2)
        if ratingx2 != old_rating:
            changed = True
-            is_rating = db.session.query(db.Ratings).filter(db.Ratings.rating == ratingx2).first()
+            is_rating = calibre_db.session.query(db.Ratings).filter(db.Ratings.rating == ratingx2).first()
            if is_rating:
                book.ratings.append(is_rating)
            else:
@@ -307,15 +317,63 @@ def edit_book_ratings(to_save, book):
            changed = True
    return changed

+def edit_book_tags(tags, book):
+    input_tags = tags.split(',')
+    input_tags = list(map(lambda it: it.strip(), input_tags))
+    # Remove duplicates
+    input_tags = helper.uniq(input_tags)
+    return modify_database_object(input_tags, book.tags, db.Tags, calibre_db.session, 'tags')

-def edit_book_languages(to_save, book):
-    input_languages = to_save["languages"].split(',')
+
+def edit_book_series(series, book):
+    input_series = [series.strip()]
+    input_series = [x for x in input_series if x != '']
+    return modify_database_object(input_series, book.series, db.Series, calibre_db.session, 'series')
+
+
+def edit_book_series_index(series_index, book):
+    # Add default series_index to book
+    modif_date = False
+    series_index = series_index or '1'
+    if book.series_index != series_index:
+        book.series_index = series_index
+        modif_date = True
+    return modif_date
+
+# Handle book comments/description
+def edit_book_comments(comments, book):
+    modif_date = False
+    if len(book.comments):
+        if book.comments[0].text != comments:
+            book.comments[0].text = comments
+            modif_date = True
+    else:
+        if comments:
+            book.comments.append(db.Comments(text=comments, book=book.id))
+            modif_date = True
+    return modif_date
+
+
+def edit_book_languages(languages, book, upload=False):
+    input_languages = languages.split(',')
    unknown_languages = []
-    input_l = isoLanguages.get_language_codes(get_locale(), input_languages, unknown_languages)
+    if not upload:
+        input_l = isoLanguages.get_language_codes(get_locale(), input_languages, unknown_languages)
+    else:
+        input_l = isoLanguages.get_valid_language_codes(get_locale(), input_languages, unknown_languages)
    for l in unknown_languages:
        log.error('%s is not a valid language', l)
-        flash(_(u"%(langname)s is not a valid language", langname=l), category="error")
-    return modify_database_object(list(input_l), book.languages, db.Languages, db.session, 'languages')
+        flash(_(u"%(langname)s is not a valid language", langname=l), category="warning")
+    # ToDo: Not working correct
+    if upload and len(input_l) == 1:
+        # If the language of the file is excluded from the users view, it's not imported, to allow the user to view
+        # the book it's language is set to the filter language
+        if input_l[0] != current_user.filter_language() and current_user.filter_language() != "all":
+            input_l[0] = calibre_db.session.query(db.Languages). \
+                filter(db.Languages.lang_code == current_user.filter_language()).first()
+    # Remove duplicates
+    input_l = helper.uniq(input_l)
+    return modify_database_object(input_l, book.languages, db.Languages, calibre_db.session, 'languages')


 def edit_book_publisher(to_save, book):
@@ -323,15 +381,16 @@ def edit_book_publisher(to_save, book):
    if to_save["publisher"]:
        publisher = to_save["publisher"].rstrip().strip()
        if len(book.publishers) == 0 or (len(book.publishers) > 0 and publisher != book.publishers[0].name):
-            changed |= modify_database_object([publisher], book.publishers, db.Publishers, db.session, 'publisher')
+            changed |= modify_database_object([publisher], book.publishers, db.Publishers, calibre_db.session,
+                                              'publisher')
    elif len(book.publishers):
-        changed |= modify_database_object([], book.publishers, db.Publishers, db.session, 'publisher')
+        changed |= modify_database_object([], book.publishers, db.Publishers, calibre_db.session, 'publisher')
    return changed


 def edit_cc_data(book_id, book, to_save):
    changed = False
-    cc = db.session.query(db.Custom_Columns).filter(db.Custom_Columns.datatype.notin_(db.cc_exceptions)).all()
+    cc = calibre_db.session.query(db.Custom_Columns).filter(db.Custom_Columns.datatype.notin_(db.cc_exceptions)).all()
    for c in cc:
        cc_string = "custom_column_" + str(c.id)
        if not c.is_multiple:
@@ -354,12 +413,12 @@ def edit_cc_data(book_id, book, to_save):
                            else:
                                del_cc = getattr(book, cc_string)[0]
                                getattr(book, cc_string).remove(del_cc)
-                                db.session.delete(del_cc)
+                                calibre_db.session.delete(del_cc)
                                changed = True
                        else:
                            cc_class = db.cc_classes[c.id]
                            new_cc = cc_class(value=to_save[cc_string], book=book_id)
-                            db.session.add(new_cc)
+                            calibre_db.session.add(new_cc)
                            changed = True

                else:
@@ -371,18 +430,18 @@ def edit_cc_data(book_id, book, to_save):
                            del_cc = getattr(book, cc_string)[0]
                            getattr(book, cc_string).remove(del_cc)
                            if len(del_cc.books) == 0:
-                                db.session.delete(del_cc)
+                                calibre_db.session.delete(del_cc)
                                changed = True
                        cc_class = db.cc_classes[c.id]
-                        new_cc = db.session.query(cc_class).filter(
+                        new_cc = calibre_db.session.query(cc_class).filter(
                            cc_class.value == to_save[cc_string].strip()).first()
                        # if no cc val is found add it
                        if new_cc is None:
                            new_cc = cc_class(value=to_save[cc_string].strip())
-                            db.session.add(new_cc)
+                            calibre_db.session.add(new_cc)
                            changed = True
-                            db.session.flush()
-                            new_cc = db.session.query(cc_class).filter(
+                            calibre_db.session.flush()
+                            new_cc = calibre_db.session.query(cc_class).filter(
                                cc_class.value == to_save[cc_string].strip()).first()
                        # add cc value to book
                        getattr(book, cc_string).append(new_cc)
@@ -392,68 +451,79 @@ def edit_cc_data(book_id, book, to_save):
                    del_cc = getattr(book, cc_string)[0]
                    getattr(book, cc_string).remove(del_cc)
                    if not del_cc.books or len(del_cc.books) == 0:
-                        db.session.delete(del_cc)
+                        calibre_db.session.delete(del_cc)
                        changed = True
        else:
            input_tags = to_save[cc_string].split(',')
            input_tags = list(map(lambda it: it.strip(), input_tags))
-            changed |= modify_database_object(input_tags, getattr(book, cc_string), db.cc_classes[c.id], db.session,
-                                   'custom')
+            changed |= modify_database_object(input_tags,
+                                              getattr(book, cc_string),
+                                              db.cc_classes[c.id],
+                                              calibre_db.session,
+                                              'custom')
    return changed

 def upload_single_file(request, book, book_id):
    # Check and handle Uploaded file
    if 'btn-upload-format' in request.files:
-        requested_file = request.files['btn-upload-format']
-        # check for empty request
-        if requested_file.filename != '':
-            if '.' in requested_file.filename:
-                file_ext = requested_file.filename.rsplit('.', 1)[-1].lower()
-                if file_ext not in constants.EXTENSIONS_UPLOAD:
-                    flash(_("File extension '%(ext)s' is not allowed to be uploaded to this server", ext=file_ext),
-                          category="error")
+            requested_file = request.files['btn-upload-format']
+            # check for empty request
+            if requested_file.filename != '':
+                if not current_user.role_upload():
+                    abort(403)
+                if '.' in requested_file.filename:
+                    file_ext = requested_file.filename.rsplit('.', 1)[-1].lower()
+                    if file_ext not in constants.EXTENSIONS_UPLOAD and '' not in constants.EXTENSIONS_UPLOAD:
+                        flash(_("File extension '%(ext)s' is not allowed to be uploaded to this server", ext=file_ext),
+                              category="error")
+                        return redirect(url_for('web.show_book', book_id=book.id))
+                else:
+                    flash(_('File to be uploaded must have an extension'), category="error")
                    return redirect(url_for('web.show_book', book_id=book.id))
-            else:
-                flash(_('File to be uploaded must have an extension'), category="error")
-                return redirect(url_for('web.show_book', book_id=book.id))

-            file_name = book.path.rsplit('/', 1)[-1]
-            filepath = os.path.normpath(os.path.join(config.config_calibre_dir, book.path))
-            saved_filename = os.path.join(filepath, file_name + '.' + file_ext)
+                file_name = book.path.rsplit('/', 1)[-1]
+                filepath = os.path.normpath(os.path.join(config.config_calibre_dir, book.path))
+                saved_filename = os.path.join(filepath, file_name + '.' + file_ext)

-            # check if file path exists, otherwise create it, copy file to calibre path and delete temp file
-            if not os.path.exists(filepath):
+                # check if file path exists, otherwise create it, copy file to calibre path and delete temp file
+                if not os.path.exists(filepath):
+                    try:
+                        os.makedirs(filepath)
+                    except OSError:
+                        flash(_(u"Failed to create path %(path)s (Permission denied).", path=filepath), category="error")
+                        return redirect(url_for('web.show_book', book_id=book.id))
                try:
-                    os.makedirs(filepath)
+                    requested_file.save(saved_filename)
                except OSError:
-                    flash(_(u"Failed to create path %(path)s (Permission denied).", path=filepath), category="error")
+                    flash(_(u"Failed to store file %(file)s.", file=saved_filename), category="error")
                    return redirect(url_for('web.show_book', book_id=book.id))
-            try:
-                requested_file.save(saved_filename)
-            except OSError:
-                flash(_(u"Failed to store file %(file)s.", file=saved_filename), category="error")
-                return redirect(url_for('web.show_book', book_id=book.id))

-            file_size = os.path.getsize(saved_filename)
-            is_format = db.session.query(db.Data).filter(db.Data.book == book_id).\
-                filter(db.Data.format == file_ext.upper()).first()
+                file_size = os.path.getsize(saved_filename)
+                is_format = calibre_db.get_book_format(book_id, file_ext.upper())

-            # Format entry already exists, no need to update the database
-            if is_format:
-                log.warning('Book format %s already existing', file_ext.upper())
-            else:
-                db_format = db.Data(book_id, file_ext.upper(), file_size, file_name)
-                db.session.add(db_format)
-                db.session.commit()
-                db.update_title_sort(config)
+                # Format entry already exists, no need to update the database
+                if is_format:
+                    log.warning('Book format %s already existing', file_ext.upper())
+                else:
+                    try:
+                        db_format = db.Data(book_id, file_ext.upper(), file_size, file_name)
+                        calibre_db.session.add(db_format)
+                        calibre_db.session.commit()
+                        calibre_db.update_title_sort(config)
+                    except OperationalError as e:
+                        calibre_db.session.rollback()
+                        log.error('Database error: %s', e)
+                        flash(_(u"Database error: %(error)s.", error=e), category="error")
+                        return redirect(url_for('web.show_book', book_id=book.id))

-            # Queue uploader info
-            uploadText=_(u"File format %(ext)s added to %(book)s", ext=file_ext.upper(), book=book.title)
-            worker.add_upload(current_user.nickname,
-                "<a href=\"" + url_for('web.show_book', book_id=book.id) + "\">" + uploadText + "</a>")
+                # Queue uploader info
+                uploadText=_(u"File format %(ext)s added to %(book)s", ext=file_ext.upper(), book=book.title)
+                worker.add_upload(current_user.nickname,
+                    "<a href=\"" + url_for('web.show_book', book_id=book.id) + "\">" + uploadText + "</a>")

-            return uploader.process(
-                saved_filename, *os.path.splitext(requested_file.filename), config.config_rarfile_location)
+                return uploader.process(
+                    saved_filename, *os.path.splitext(requested_file.filename),
+                    rarExecutable=config.config_rarfile_location)


 def upload_cover(request, book):
@@ -461,6 +531,8 @@ def upload_cover(request, book):
        requested_file = request.files['btn-upload-cover']
        # check for empty request
        if requested_file.filename != '':
+            if not current_user.role_upload():
+                abort(403)
            ret, message = helper.save_cover(requested_file, book.path)
            if ret is True:
                return True
@@ -480,9 +552,8 @@ def edit_book(book_id):
        return render_edit_book(book_id)

    # create the function for sorting...
-    db.update_title_sort(config)
-    book = db.session.query(db.Books)\
-        .filter(db.Books.id == book_id).filter(common_filters()).first()
+    calibre_db.update_title_sort(config)
+    book = calibre_db.get_filtered_book(book_id)

    # Book not found
    if not book:
@@ -509,17 +580,19 @@ def edit_book(book_id):
        # handle author(s)
        input_authors = to_save["author_name"].split('&')
        input_authors = list(map(lambda it: it.strip().replace(',', '|'), input_authors))
+        # Remove duplicates in authors list
+        input_authors = helper.uniq(input_authors)
        # we have all author names now
        if input_authors == ['']:
            input_authors = [_(u'Unknown')]  # prevent empty Author

-        modif_date |= modify_database_object(input_authors, book.authors, db.Authors, db.session, 'author')
+        modif_date |= modify_database_object(input_authors, book.authors, db.Authors, calibre_db.session, 'author')

        # Search for each author if author is in database, if not, authorname and sorted authorname is generated new
        # everything then is assembled for sorted author field in database
        sort_authors_list = list()
        for inp in input_authors:
-            stored_author = db.session.query(db.Authors).filter(db.Authors.name == inp).first()
+            stored_author = calibre_db.session.query(db.Authors).filter(db.Authors.name == inp).first()
            if not stored_author:
                stored_author = helper.get_sorted_author(inp)
            else:
@@ -539,41 +612,34 @@ def edit_book(book_id):
            error = helper.update_dir_stucture(edited_books_id, config.config_calibre_dir, input_authors[0])

        if not error:
-            if to_save["cover_url"]:
-                result, error = helper.save_cover_from_url(to_save["cover_url"], book.path)
-                if result is True:
-                    book.has_cover = 1
-                    modif_date = True
-                else:
-                    flash(error, category="error")
+            if "cover_url" in to_save:
+                if to_save["cover_url"]:
+                    if not current_user.role_upload():
+                        return "", (403)
+                    result, error = helper.save_cover_from_url(to_save["cover_url"], book.path)
+                    if result is True:
+                        book.has_cover = 1
+                        modif_date = True
+                    else:
+                        flash(error, category="error")

-            if book.series_index != to_save["series_index"]:
-                book.series_index = to_save["series_index"]
-                modif_date = True
+            # Add default series_index to book
+            modif_date |= edit_book_series_index(to_save["series_index"], book)

            # Handle book comments/description
-            if len(book.comments):
-                if book.comments[0].text != to_save["description"]:
-                    book.comments[0].text = to_save["description"]
-                    modif_date = True
-            else:
-                if to_save["description"]:
-                    book.comments.append(db.Comments(text=to_save["description"], book=book.id))
-                    modif_date = True
+            modif_date |= edit_book_comments(to_save["description"], book)

-                    # Handle identifiers
+            # Handle identifiers
            input_identifiers = identifier_list(to_save, book)
-            modif_date |= modify_identifiers(input_identifiers, book.identifiers, db.session)
-
+            modification, warning = modify_identifiers(input_identifiers, book.identifiers, calibre_db.session)
+            if warning:
+                flash(_("Identifiers are not Case Sensitive, Overwriting Old Identifier"), category="warning")
+            modif_date |= modification
            # Handle book tags
-            input_tags = to_save["tags"].split(',')
-            input_tags = list(map(lambda it: it.strip(), input_tags))
-            modif_date |= modify_database_object(input_tags, book.tags, db.Tags, db.session, 'tags')
+            modif_date |= edit_book_tags(to_save['tags'], book)

            # Handle book series
-            input_series = [to_save["series"].strip()]
-            input_series = [x for x in input_series if x != '']
-            modif_date |= modify_database_object(input_series, book.series, db.Series, db.session, 'series')
+            modif_date |= edit_book_series(to_save["series"], book)

            if to_save["pubdate"]:
                try:
@@ -587,7 +653,7 @@ def edit_book(book_id):
            modif_date |= edit_book_publisher(to_save, book)

            # handle book languages
-            modif_date |= edit_book_languages(to_save, book)
+            modif_date |= edit_book_languages(to_save['languages'], book)

            # handle book ratings
            modif_date |= edit_book_ratings(to_save, book)
@@ -597,7 +663,8 @@ def edit_book(book_id):

            if modif_date:
                book.last_modified = datetime.utcnow()
-            db.session.commit()
+            calibre_db.session.merge(book)
+            calibre_db.session.commit()
            if config.config_use_google_drive:
                gdriveutils.updateGdriveCalibreFromLocal()
            if "detail_view" in to_save:
@@ -606,12 +673,12 @@ def edit_book(book_id):
                flash(_("Metadata successfully updated"), category="success")
                return render_edit_book(book_id)
        else:
-            db.session.rollback()
+            calibre_db.session.rollback()
            flash(error, category="error")
            return render_edit_book(book_id)
    except Exception as e:
        log.exception(e)
-        db.session.rollback()
+        calibre_db.session.rollback()
        flash(_("Error editing book, please check logfile for details"), category="error")
        return redirect(url_for('web.show_book', book_id=book.id))

@@ -640,7 +707,7 @@ def identifier_list(to_save, book):
        val_key = id_val_prefix + type_key[len(id_type_prefix):]
        if val_key not in to_save.keys():
            continue
-        result.append( db.Identifiers(to_save[val_key], type_value, book.id) )
+        result.append(db.Identifiers(to_save[val_key], type_value, book.id))
    return result

@editbook.route("/upload", methods=["GET", "POST"])
@@ -651,179 +718,175 @@ def upload():
        abort(404)
    if request.method == 'POST' and 'btn-upload' in request.files:
        for requested_file in request.files.getlist("btn-upload"):
-            # create the function for sorting...
-            db.update_title_sort(config)
-            db.session.connection().connection.connection.create_function('uuid4', 0, lambda: str(uuid4()))
-
-            # check if file extension is correct
-            if '.' in requested_file.filename:
-                file_ext = requested_file.filename.rsplit('.', 1)[-1].lower()
-                if file_ext not in constants.EXTENSIONS_UPLOAD:
-                    flash(
-                        _("File extension '%(ext)s' is not allowed to be uploaded to this server",
-                          ext=file_ext), category="error")
-                    return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')
-            else:
-                flash(_('File to be uploaded must have an extension'), category="error")
-                return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')
-
-            # extract metadata from file
            try:
-                meta = uploader.upload(requested_file, config.config_rarfile_location)
-            except (IOError, OSError):
-                log.error("File %s could not saved to temp dir", requested_file.filename)
-                flash(_(u"File %(filename)s could not saved to temp dir",
-                        filename= requested_file.filename), category="error")
-                return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')
-            title = meta.title
-            authr = meta.author
-            tags = meta.tags
-            series = meta.series
-            series_index = meta.series_id
-            title_dir = helper.get_valid_filename(title)
-            author_dir = helper.get_valid_filename(authr)
-            filepath = os.path.join(config.config_calibre_dir, author_dir, title_dir)
-            saved_filename = os.path.join(filepath, title_dir + meta.extension.lower())
+                modif_date = False
+                # create the function for sorting...
+                calibre_db.update_title_sort(config)
+                calibre_db.session.connection().connection.connection.create_function('uuid4', 0, lambda: str(uuid4()))

-            if title != _(u'Unknown') and authr != _(u'Unknown'):
-                entry = helper.check_exists_book(authr, title)
-                if entry:
-                    log.info("Uploaded book probably exists in library")
-                    flash(_(u"Uploaded book probably exists in the library, consider to change before upload new: ")
-                        + Markup(render_title_template('book_exists_flash.html', entry=entry)), category="warning")
-
-            # check if file path exists, otherwise create it, copy file to calibre path and delete temp file
-            if not os.path.exists(filepath):
-                try:
-                    os.makedirs(filepath)
-                except OSError:
-                    log.error("Failed to create path %s (Permission denied)", filepath)
-                    flash(_(u"Failed to create path %(path)s (Permission denied).", path=filepath), category="error")
+                # check if file extension is correct
+                if '.' in requested_file.filename:
+                    file_ext = requested_file.filename.rsplit('.', 1)[-1].lower()
+                    if file_ext not in constants.EXTENSIONS_UPLOAD and '' not in constants.EXTENSIONS_UPLOAD:
+                        flash(
+                            _("File extension '%(ext)s' is not allowed to be uploaded to this server",
+                              ext=file_ext), category="error")
+                        return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')
+                else:
+                    flash(_('File to be uploaded must have an extension'), category="error")
                    return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')
-            try:
-                copyfile(meta.file_path, saved_filename)
-                os.unlink(meta.file_path)
-            except OSError as e:
-                log.error("Failed to move file %s: %s", saved_filename, e)
-                flash(_(u"Failed to Move File %(file)s: %(error)s", file=saved_filename, error=e), category="error")
-                return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')

-            if meta.cover is None:
-                has_cover = 0
-                copyfile(os.path.join(constants.STATIC_DIR, 'generic_cover.jpg'),
-                         os.path.join(filepath, "cover.jpg"))
-            else:
-                has_cover = 1
+                # extract metadata from file
                try:
-                    copyfile(meta.cover, os.path.join(filepath, "cover.jpg"))
-                    os.unlink(meta.cover)
+                    meta = uploader.upload(requested_file, config.config_rarfile_location)
+                except (IOError, OSError):
+                    log.error("File %s could not saved to temp dir", requested_file.filename)
+                    flash(_(u"File %(filename)s could not saved to temp dir",
+                            filename= requested_file.filename), category="error")
+                    return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')
+                title = meta.title
+                authr = meta.author
+
+                if title != _(u'Unknown') and authr != _(u'Unknown'):
+                    entry = calibre_db.check_exists_book(authr, title)
+                    if entry:
+                        log.info("Uploaded book probably exists in library")
+                        flash(_(u"Uploaded book probably exists in the library, consider to change before upload new: ")
+                            + Markup(render_title_template('book_exists_flash.html', entry=entry)), category="warning")
+
+                # handle authors
+                input_authors = authr.split('&')
+                # handle_authors(input_authors)
+                input_authors = list(map(lambda it: it.strip().replace(',', '|'), input_authors))
+                # Remove duplicates in authors list
+                input_authors = helper.uniq(input_authors)
+
+                # we have all author names now
+                if input_authors == ['']:
+                    input_authors = [_(u'Unknown')]  # prevent empty Author
+
+                sort_authors_list=list()
+                db_author = None
+                for inp in input_authors:
+                    stored_author = calibre_db.session.query(db.Authors).filter(db.Authors.name == inp).first()
+                    if not stored_author:
+                        if not db_author:
+                            db_author = db.Authors(inp, helper.get_sorted_author(inp), "")
+                            calibre_db.session.add(db_author)
+                            calibre_db.session.commit()
+                        sort_author = helper.get_sorted_author(inp)
+                    else:
+                        if not db_author:
+                            db_author = stored_author
+                        sort_author = stored_author.sort
+                    sort_authors_list.append(sort_author) # helper.get_sorted_author(sort_author))
+                sort_authors = ' & '.join(sort_authors_list)
+
+                title_dir = helper.get_valid_filename(title)
+                author_dir = helper.get_valid_filename(db_author.name)
+                filepath = os.path.join(config.config_calibre_dir, author_dir, title_dir)
+                saved_filename = os.path.join(filepath, title_dir + meta.extension.lower())
+
+                # check if file path exists, otherwise create it, copy file to calibre path and delete temp file
+                if not os.path.exists(filepath):
+                    try:
+                        os.makedirs(filepath)
+                    except OSError:
+                        log.error("Failed to create path %s (Permission denied)", filepath)
+                        flash(_(u"Failed to create path %(path)s (Permission denied).", path=filepath), category="error")
+                        return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')
+                try:
+                    copyfile(meta.file_path, saved_filename)
+                    os.unlink(meta.file_path)
                except OSError as e:
-                    log.error("Failed to move cover file %s: %s", os.path.join(filepath, "cover.jpg"), e)
-                    flash(_(u"Failed to Move Cover File %(file)s: %(error)s", file=os.path.join(filepath, "cover.jpg"),
-                            error=e),
-                          category="error")
+                    log.error("Failed to move file %s: %s", saved_filename, e)
+                    flash(_(u"Failed to Move File %(file)s: %(error)s", file=saved_filename, error=e), category="error")
                    return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')

-            # handle authors
-            is_author = db.session.query(db.Authors).filter(db.Authors.name == authr).first()
-            if is_author:
-                db_author = is_author
-            else:
-                db_author = db.Authors(authr, helper.get_sorted_author(authr), "")
-                db.session.add(db_author)
-
-            # handle series
-            db_series = None
-            is_series = db.session.query(db.Series).filter(db.Series.name == series).first()
-            if is_series:
-                db_series = is_series
-            elif series != '':
-                db_series = db.Series(series, "")
-                db.session.add(db_series)
-
-            # add language actually one value in list
-            input_language = meta.languages
-            db_language = None
-            if input_language != "":
-                input_language = isoLanguages.get(name=input_language).part3
-                hasLanguage = db.session.query(db.Languages).filter(db.Languages.lang_code == input_language).first()
-                if hasLanguage:
-                    db_language = hasLanguage
+                if meta.cover is None:
+                    has_cover = 0
+                    copyfile(os.path.join(constants.STATIC_DIR, 'generic_cover.jpg'),
+                             os.path.join(filepath, "cover.jpg"))
                else:
-                    db_language = db.Languages(input_language)
-                    db.session.add(db_language)
+                    has_cover = 1

-            # If the language of the file is excluded from the users view, it's not imported, to allow the user to view
-            # the book it's language is set to the filter language
-            if db_language != current_user.filter_language() and current_user.filter_language() != "all":
-                db_language = db.session.query(db.Languages).\
-                    filter(db.Languages.lang_code == current_user.filter_language()).first()
+                # combine path and normalize path from windows systems
+                path = os.path.join(author_dir, title_dir).replace('\\', '/')
+                # Calibre adds books with utc as timezone
+                db_book = db.Books(title, "", sort_authors, datetime.utcnow(), datetime(101, 1, 1),
+                                   '1', datetime.utcnow(), path, has_cover, db_author, [], "")

-            # combine path and normalize path from windows systems
-            path = os.path.join(author_dir, title_dir).replace('\\', '/')
-            # Calibre adds books with utc as timezone
-            db_book = db.Books(title, "", db_author.sort, datetime.utcnow(), datetime(101, 1, 1),
-                               series_index, datetime.utcnow(), path, has_cover, db_author, [], db_language)
-            db_book.authors.append(db_author)
-            if db_series:
-                db_book.series.append(db_series)
-            if db_language is not None:
-                db_book.languages.append(db_language)
-            file_size = os.path.getsize(saved_filename)
-            db_data = db.Data(db_book, meta.extension.upper()[1:], file_size, title_dir)
+                modif_date |= modify_database_object(input_authors, db_book.authors, db.Authors, calibre_db.session,
+                                                     'author')

-            # handle tags
-            input_tags = tags.split(',')
-            input_tags = list(map(lambda it: it.strip(), input_tags))
-            if input_tags[0] !="":
-                modify_database_object(input_tags, db_book.tags, db.Tags, db.session, 'tags')
+                # Add series_index to book
+                modif_date |= edit_book_series_index(meta.series_id, db_book)

-            # flush content, get db_book.id available
-            db_book.data.append(db_data)
-            db.session.add(db_book)
-            db.session.flush()
+                # add languages
+                modif_date |= edit_book_languages(meta.languages, db_book, upload=True)

-            # add comment
-            book_id = db_book.id
-            upload_comment = Markup(meta.description).unescape()
-            if upload_comment != "":
-                db.session.add(db.Comments(upload_comment, book_id))
+                # handle tags
+                modif_date |= edit_book_tags(meta.tags, db_book)

-            # save data to database, reread data
-            db.session.commit()
-            db.update_title_sort(config)
-            # Reread book. It's important not to filter the result, as it could have language which hide it from
-            # current users view (tags are not stored/extracted from metadata and could also be limited)
-            book = db.session.query(db.Books).filter(db.Books.id == book_id).first()
-            # upload book to gdrive if nesseccary and add "(bookid)" to folder name
-            if config.config_use_google_drive:
-                gdriveutils.updateGdriveCalibreFromLocal()
-            error = helper.update_dir_stucture(book.id, config.config_calibre_dir)
-            db.session.commit()
-            if config.config_use_google_drive:
-                gdriveutils.updateGdriveCalibreFromLocal()
-            if error:
-                flash(error, category="error")
-            uploadText=_(u"File %(file)s uploaded", file=book.title)
-            worker.add_upload(current_user.nickname,
-                "<a href=\"" + url_for('web.show_book', book_id=book.id) + "\">" + uploadText + "</a>")
+                # handle series
+                modif_date |= edit_book_series(meta.series, db_book)

-            # create data for displaying display Full language name instead of iso639.part3language
-            if db_language is not None:
-                book.languages[0].language_name = _(meta.languages)
-            author_names = []
-            for author in db_book.authors:
-                author_names.append(author.name)
-            if len(request.files.getlist("btn-upload")) < 2:
-                if current_user.role_edit() or current_user.role_admin():
-                    resp = {"location": url_for('editbook.edit_book', book_id=db_book.id)}
-                    return Response(json.dumps(resp), mimetype='application/json')
-                else:
-                    resp = {"location": url_for('web.show_book', book_id=db_book.id)}
-                    return Response(json.dumps(resp), mimetype='application/json')
+                # Add file to book
+                file_size = os.path.getsize(saved_filename)
+                db_data = db.Data(db_book, meta.extension.upper()[1:], file_size, title_dir)
+                db_book.data.append(db_data)
+                calibre_db.session.add(db_book)
+
+                # flush content, get db_book.id available
+                calibre_db.session.flush()
+
+                # Comments needs book id therfore only possiblw after flush
+                modif_date |= edit_book_comments(Markup(meta.description).unescape(), db_book)
+
+                book_id = db_book.id
+                title = db_book.title
+
+                error = helper.update_dir_stucture(book_id, config.config_calibre_dir, input_authors[0])
+
+                # move cover to final directory, including book id
+                if has_cover:
+                    new_coverpath = os.path.join(config.config_calibre_dir, db_book.path, "cover.jpg")
+                    try:
+                        copyfile(meta.cover, new_coverpath)
+                        os.unlink(meta.cover)
+                    except OSError as e:
+                        log.error("Failed to move cover file %s: %s", new_coverpath, e)
+                        flash(_(u"Failed to Move Cover File %(file)s: %(error)s", file=new_coverpath,
+                                error=e),
+                              category="error")
+
+                # save data to database, reread data
+                calibre_db.session.commit()
+                #calibre_db.setup_db(config, ub.app_DB_path)
+                # Reread book. It's important not to filter the result, as it could have language which hide it from
+                # current users view (tags are not stored/extracted from metadata and could also be limited)
+                #book = calibre_db.get_book(book_id)
+                if config.config_use_google_drive:
+                    gdriveutils.updateGdriveCalibreFromLocal()
+                if error:
+                    flash(error, category="error")
+                uploadText=_(u"File %(file)s uploaded", file=title)
+                worker.add_upload(current_user.nickname,
+                    "<a href=\"" + url_for('web.show_book', book_id=book_id) + "\">" + uploadText + "</a>")
+
+                if len(request.files.getlist("btn-upload")) < 2:
+                    if current_user.role_edit() or current_user.role_admin():
+                        resp = {"location": url_for('editbook.edit_book', book_id=book_id)}
+                        return Response(json.dumps(resp), mimetype='application/json')
+                    else:
+                        resp = {"location": url_for('web.show_book', book_id=book_id)}
+                        return Response(json.dumps(resp), mimetype='application/json')
+            except OperationalError as e:
+                calibre_db.session.rollback()
+                log.error("Database error: %s", e)
+                flash(_(u"Database error: %(error)s.", error=e), category="error")
        return Response(json.dumps({"location": url_for("web.index")}), mimetype='application/json')

-
@editbook.route("/admin/book/convert/<int:book_id>", methods=['POST'])
@login_required_if_no_ano
@edit_required
@@ -22,6 +22,7 @@ import zipfile
 from lxml import etree

 from . import isoLanguages
+from .helper import split_authors
 from .constants import BookMeta


@@ -64,9 +65,9 @@ def get_epub_info(tmp_file_path, original_file_name, original_file_extension):
        tmp = p.xpath('dc:%s/text()' % s, namespaces=ns)
        if len(tmp) > 0:
            if s == 'creator':
-                 epub_metadata[s] = ' & '.join(p.xpath('dc:%s/text()' % s, namespaces=ns))
+                epub_metadata[s] = ' & '.join(split_authors(p.xpath('dc:%s/text()' % s, namespaces=ns)))
            elif s == 'subject':
-                 epub_metadata[s] = ', '.join(p.xpath('dc:%s/text()' % s, namespaces=ns))
+                epub_metadata[s] = ', '.join(p.xpath('dc:%s/text()' % s, namespaces=ns))
            else:
                epub_metadata[s] = p.xpath('dc:%s/text()' % s, namespaces=ns)[0]
        else:
@@ -82,16 +83,8 @@ def get_epub_info(tmp_file_path, original_file_name, original_file_extension):
        else:
            epub_metadata['description'] = ""

-    if epub_metadata['language'] == u'Unknown':
-        epub_metadata['language'] = ""
-    else:
-        lang = epub_metadata['language'].split('-', 1)[0].lower()
-        if len(lang) == 2:
-            epub_metadata['language'] = isoLanguages.get(part1=lang).name
-        elif len(lang) == 3:
-            epub_metadata['language'] = isoLanguages.get(part3=lang).name
-        else:
-            epub_metadata['language'] = ""
+    lang = epub_metadata['language'].split('-', 1)[0].lower()
+    epub_metadata['language'] = isoLanguages.get_lang3(lang)

    series = tree.xpath("/pkg:package/pkg:metadata/pkg:meta[@name='calibre:series']/@content", namespaces=ns)
    if len(series) > 0:
@@ -122,9 +115,14 @@ def get_epub_info(tmp_file_path, original_file_name, original_file_extension):
                markupTree = etree.fromstring(markup)
                # no matter xhtml or html with no namespace
                imgsrc = markupTree.xpath("//*[local-name() = 'img']/@src")
-                # imgsrc maybe startwith "../"" so fullpath join then relpath to cwd
-                filename = os.path.relpath(os.path.join(os.path.dirname(os.path.join(coverpath, coversection[0])), imgsrc[0]))
-                coverfile = extractCover(epubZip, filename, "", tmp_file_path)
+                # Alternative image source
+                if not len(imgsrc):
+                    imgsrc = markupTree.xpath("//attribute::*[contains(local-name(), 'href')]")
+                if len(imgsrc):
+                    # imgsrc maybe startwith "../"" so fullpath join then relpath to cwd
+                    filename = os.path.relpath(os.path.join(os.path.dirname(os.path.join(coverpath, coversection[0])),
+                                                            imgsrc[0]))
+                    coverfile = extractCover(epubZip, filename, "", tmp_file_path)
            else:
                coverfile = extractCover(epubZip, coversection[0], coverpath, tmp_file_path)

@@ -34,18 +34,17 @@ from flask import Blueprint, flash, request, redirect, url_for, abort
 from flask_babel import gettext as _
 from flask_login import login_required

-try:
-    from googleapiclient.errors import HttpError
-except ImportError:
-    pass
-
-from . import logger, gdriveutils, config, db
+from . import logger, gdriveutils, config, ub, calibre_db
 from .web import admin_required

-
 gdrive = Blueprint('gdrive', __name__)
 log = logger.create()

+try:
+    from googleapiclient.errors import HttpError
+except ImportError as err:
+    log.debug("Cannot import googleapiclient, using GDrive will not work: %s", err)
+
 current_milli_time = lambda: int(round(time() * 1000))

 gdrive_watch_callback_token = 'target=calibreweb-watch_files'
@@ -73,7 +72,7 @@ def google_drive_callback():
        credentials = gdriveutils.Gauth.Instance().auth.flow.step2_exchange(auth_code)
        with open(gdriveutils.CREDENTIALS, 'w') as f:
            f.write(credentials.to_json())
-    except ValueError as error:
+    except (ValueError, AttributeError) as error:
        log.error(error)
    return redirect(url_for('admin.configuration'))

@@ -94,8 +93,7 @@ def watch_gdrive():
        try:
            result = gdriveutils.watchChange(gdriveutils.Gdrive.Instance().drive, notification_id,
                               'web_hook', address, gdrive_watch_callback_token, current_milli_time() + 604800*1000)
-            config.config_google_drive_watch_changes_response = json.dumps(result)
-            # after save(), config_google_drive_watch_changes_response will be a json object, not string
+            config.config_google_drive_watch_changes_response = result
            config.save()
        except HttpError as e:
            reason=json.loads(e.content)['error']['errors'][0]
@@ -118,7 +116,7 @@ def revoke_watch_gdrive():
                                    last_watch_response['resourceId'])
        except HttpError:
            pass
-        config.config_google_drive_watch_changes_response = None
+        config.config_google_drive_watch_changes_response = {}
        config.save()
    return redirect(url_for('admin.configuration'))

@@ -145,7 +143,8 @@ def on_received_watch_confirmation():
                        dbpath = os.path.join(config.config_calibre_dir, "metadata.db")
                    else:
                        dbpath = os.path.join(config.config_calibre_dir, "metadata.db").encode()
-                    if not response['deleted'] and response['file']['title'] == 'metadata.db' and response['file']['md5Checksum'] != hashlib.md5(dbpath):
+                    if not response['deleted'] and response['file']['title'] == 'metadata.db' \
+                       and response['file']['md5Checksum'] != hashlib.md5(dbpath):
                        tmpDir = tempfile.gettempdir()
                        log.info('Database file updated')
                        copyfile(dbpath, os.path.join(tmpDir, "metadata.db_" + str(current_milli_time())))
@@ -154,7 +153,7 @@ def on_received_watch_confirmation():
                        log.info('Setting up new DB')
                        # prevent error on windows, as os.rename does on exisiting files
                        move(os.path.join(tmpDir, "tmp_metadata.db"), dbpath)
-                        db.setup_db(config)
+                        calibre_db.reconnect_db(config, ub.app_DB_path)
            except Exception as e:
                log.exception(e)
        updateMetaData()
@@ -27,14 +27,18 @@ from sqlalchemy import Column, UniqueConstraint
 from sqlalchemy import String, Integer
 from sqlalchemy.orm import sessionmaker, scoped_session
 from sqlalchemy.ext.declarative import declarative_base
+from sqlalchemy.exc import OperationalError, InvalidRequestError

 try:
    from pydrive.auth import GoogleAuth
    from pydrive.drive import GoogleDrive
    from pydrive.auth import RefreshError
    from apiclient import errors
+    from httplib2 import ServerNotFoundError
    gdrive_support = True
-except ImportError:
+    importError = None
+except ImportError as err:
+    importError = err
    gdrive_support = False

 from . import logger, cli, config
@@ -50,6 +54,8 @@ if gdrive_support:
    logger.get('googleapiclient.discovery_cache').setLevel(logger.logging.ERROR)
    if not logger.is_debug_enabled():
        logger.get('googleapiclient.discovery').setLevel(logger.logging.ERROR)
+else:
+    log.debug("Cannot import pydrive,httplib2, using gdrive will not work: %s", importError)


 class Singleton:
@@ -97,7 +103,11 @@ class Singleton:
@Singleton
 class Gauth:
    def __init__(self):
-        self.auth = GoogleAuth(settings_file=SETTINGS_YAML)
+        try:
+            self.auth = GoogleAuth(settings_file=SETTINGS_YAML)
+        except NameError as error:
+            log.error(error)
+            self.auth = None


@Singleton
@@ -192,14 +202,18 @@ def getDrive(drive=None, gauth=None):
    return drive

 def listRootFolders():
-    drive = getDrive(Gdrive.Instance().drive)
-    folder = "'root' in parents and mimeType = 'application/vnd.google-apps.folder' and trashed = false"
-    fileList = drive.ListFile({'q': folder}).GetList()
+    try:
+        drive = getDrive(Gdrive.Instance().drive)
+        folder = "'root' in parents and mimeType = 'application/vnd.google-apps.folder' and trashed = false"
+        fileList = drive.ListFile({'q': folder}).GetList()
+    except ServerNotFoundError as e:
+        log.info("GDrive Error %s" % e)
+        fileList = []
    return fileList


 def getEbooksFolder(drive):
-    return getFolderInFolder('root',config.config_google_drive_folder,drive)
+    return getFolderInFolder('root', config.config_google_drive_folder, drive)


 def getFolderInFolder(parentId, folderName, drive):
@@ -229,7 +243,7 @@ def getEbooksFolderId(drive=None):
        gDriveId.path = '/'
        session.merge(gDriveId)
        session.commit()
-        return
+        return gDriveId.gdrive_id


 def getFile(pathId, fileName, drive):
@@ -474,8 +488,13 @@ def getChangeById (drive, change_id):

 # Deletes the local hashes database to force search for new folder names
 def deleteDatabaseOnChange():
-    session.query(GdriveId).delete()
-    session.commit()
+    try:
+        session.query(GdriveId).delete()
+        session.commit()
+    except (OperationalError, InvalidRequestError):
+        session.rollback()
+        log.info(u"GDrive DB is not Writeable")
+

 def updateGdriveCalibreFromLocal():
    copyToDrive(Gdrive.Instance().drive, config.config_calibre_dir, False, True)
@@ -583,8 +602,12 @@ def get_error_text(client_secrets=None):
    if not os.path.isfile(CLIENT_SECRETS):
        return 'client_secrets.json is missing or not readable'

-    with open(CLIENT_SECRETS, 'r') as settings:
-        filedata = json.load(settings)
+    try:
+        with open(CLIENT_SECRETS, 'r') as settings:
+            filedata = json.load(settings)
+    except PermissionError:
+        return 'client_secrets.json is missing or not readable'
+
    if 'web' not in filedata:
        return 'client_secrets.json is not configured for web application'
    if 'redirect_uris' not in filedata['web']:
@@ -21,9 +21,7 @@ from __future__ import division, print_function, unicode_literals
 import sys
 import os
 import io
-import json
 import mimetypes
-import random
 import re
 import shutil
 import time
@@ -32,16 +30,15 @@ from datetime import datetime, timedelta
 from tempfile import gettempdir

 import requests
-from babel import Locale as LC
-from babel.core import UnknownLocaleError
 from babel.dates import format_datetime
 from babel.units import format_unit
 from flask import send_from_directory, make_response, redirect, abort
 from flask_babel import gettext as _
 from flask_login import current_user
-from sqlalchemy.sql.expression import true, false, and_, or_, text, func
+from sqlalchemy.sql.expression import true, false, and_, text, func
 from werkzeug.datastructures import Headers
 from werkzeug.security import generate_password_hash
+from . import calibre_db

 try:
    from urllib.parse import quote
@@ -56,14 +53,14 @@ except ImportError:

 try:
    from PIL import Image as PILImage
+    from PIL import UnidentifiedImageError
    use_PIL = True
 except ImportError:
    use_PIL = False

-from . import logger, config, get_locale, db, ub, isoLanguages, worker
+from . import logger, config, get_locale, db, ub, worker
 from . import gdriveutils as gd
 from .constants import STATIC_DIR as _STATIC_DIR
-from .pagination import Pagination
 from .subproc_wrapper import process_wait
 from .worker import STAT_WAITING, STAT_FAIL, STAT_STARTED, STAT_FINISH_SUCCESS
 from .worker import TASK_EMAIL, TASK_CONVERT, TASK_UPLOAD, TASK_CONVERT_ANY
@@ -74,8 +71,8 @@ log = logger.create()

 # Convert existing book entry to new format
 def convert_book_format(book_id, calibrepath, old_book_format, new_book_format, user_id, kindle_mail=None):
-    book = db.session.query(db.Books).filter(db.Books.id == book_id).first()
-    data = db.session.query(db.Data).filter(db.Data.book == book.id).filter(db.Data.format == old_book_format).first()
+    book = calibre_db.get_book(book_id)
+    data = calibre_db.get_book_format(book.id, old_book_format)
    if not data:
        error_message = _(u"%(format)s format not found for book id: %(book)d", format=old_book_format, book=book_id)
        log.error("convert_book_format: %s", error_message)
@@ -101,10 +98,10 @@ def convert_book_format(book_id, calibrepath, old_book_format, new_book_format,
            # text = _(u"%(format)s: %(book)s", format=new_book_format, book=book.title)
        else:
            settings = dict()
-        text = (u"%s -> %s: %s" % (old_book_format, new_book_format, book.title))
+        txt = (u"%s -> %s: %s" % (old_book_format, new_book_format, book.title))
        settings['old_book_format'] = old_book_format
        settings['new_book_format'] = new_book_format
-        worker.add_convert(file_path, book.id, user_id, text, settings, kindle_mail)
+        worker.add_convert(file_path, book.id, user_id, txt, settings, kindle_mail)
        return None
    else:
        error_message = _(u"%(format)s not found: %(fn)s",
@@ -142,25 +139,27 @@ def check_send_to_kindle(entry):
    """
    if len(entry.data):
        bookformats = list()
-        if config.config_ebookconverter == 0:
+        if not config.config_converterpath:
            # no converter - only for mobi and pdf formats
            for ele in iter(entry.data):
-                if 'MOBI' in ele.format:
-                    bookformats.append({'format': 'Mobi',
-                                        'convert': 0,
-                                        'text': _('Send %(format)s to Kindle', format='Mobi')})
-                if 'PDF' in ele.format:
-                    bookformats.append({'format': 'Pdf',
-                                        'convert': 0,
-                                        'text': _('Send %(format)s to Kindle', format='Pdf')})
-                if 'AZW' in ele.format:
-                    bookformats.append({'format': 'Azw',
-                                        'convert': 0,
-                                        'text': _('Send %(format)s to Kindle', format='Azw')})
+                if ele.uncompressed_size < config.mail_size:
+                    if 'MOBI' in ele.format:
+                        bookformats.append({'format': 'Mobi',
+                                            'convert': 0,
+                                            'text': _('Send %(format)s to Kindle', format='Mobi')})
+                    if 'PDF' in ele.format:
+                        bookformats.append({'format': 'Pdf',
+                                            'convert': 0,
+                                            'text': _('Send %(format)s to Kindle', format='Pdf')})
+                    if 'AZW' in ele.format:
+                        bookformats.append({'format': 'Azw',
+                                            'convert': 0,
+                                            'text': _('Send %(format)s to Kindle', format='Azw')})
        else:
            formats = list()
            for ele in iter(entry.data):
-                formats.append(ele.format)
+                if ele.uncompressed_size < config.mail_size:
+                    formats.append(ele.format)
            if 'MOBI' in formats:
                bookformats.append({'format': 'Mobi',
                                    'convert': 0,
@@ -173,14 +172,13 @@ def check_send_to_kindle(entry):
                bookformats.append({'format': 'Pdf',
                                    'convert': 0,
                                    'text': _('Send %(format)s to Kindle', format='Pdf')})
-            if config.config_ebookconverter >= 1:
+            if config.config_converterpath:
                if 'EPUB' in formats and not 'MOBI' in formats:
                    bookformats.append({'format': 'Mobi',
                                        'convert':1,
                                        'text': _('Convert %(orig)s to %(format)s and send to Kindle',
                                                  orig='Epub',
                                                  format='Mobi')})
-            if config.config_ebookconverter == 2:
                if 'AZW3' in formats and not 'MOBI' in formats:
                    bookformats.append({'format': 'Mobi',
                                        'convert': 2,
@@ -211,7 +209,7 @@ def check_read_formats(entry):
 # 3: If Pdf file is existing, it's directly send to kindle email
 def send_mail(book_id, book_format, convert, kindle_mail, calibrepath, user_id):
    """Send email with attachments"""
-    book = db.session.query(db.Books).filter(db.Books.id == book_id).first()
+    book = calibre_db.get_book(book_id)

    if convert == 1:
        # returns None if success, otherwise errormessage
@@ -239,22 +237,22 @@ def get_valid_filename(value, replace_whitespace=True):
        value = value[:-1]+u'_'
    value = value.replace("/", "_").replace(":", "_").strip('\0')
    if use_unidecode:
-        value = (unidecode.unidecode(value)).strip()
+        value = (unidecode.unidecode(value))
    else:
        value = value.replace(u'§', u'SS')
        value = value.replace(u'ß', u'ss')
        value = unicodedata.normalize('NFKD', value)
        re_slugify = re.compile(r'[\W\s-]', re.UNICODE)
        if isinstance(value, str):  # Python3 str, Python2 unicode
-            value = re_slugify.sub('', value).strip()
+            value = re_slugify.sub('', value)
        else:
-            value = unicode(re_slugify.sub('', value).strip())
+            value = unicode(re_slugify.sub('', value))
    if replace_whitespace:
        #  *+:\"/<>? are replaced by _
-        value = re.sub(r'[\*\+:\\\"/<>\?]+', u'_', value, flags=re.U)
+        value = re.sub(r'[*+:\\\"/<>?]+', u'_', value, flags=re.U)
        # pipe has to be replaced with comma
-        value = re.sub(r'[\|]+', u',', value, flags=re.U)
-    value = value[:128]
+        value = re.sub(r'[|]+', u',', value, flags=re.U)
+    value = value[:128].strip()
    if not value:
        raise ValueError("Filename cannot be empty")
    if sys.version_info.major == 3:
@@ -263,6 +261,22 @@ def get_valid_filename(value, replace_whitespace=True):
        return value.decode('utf-8')


+def split_authors(values):
+    authors_list = []
+    for value in values:
+        authors = re.split('[&;]', value)
+        for author in authors:
+            commas = author.count(',')
+            if commas == 1:
+                author_split = author.split(',')
+                authors_list.append(author_split[1].strip() + ' ' + author_split[0].strip())
+            elif commas > 1:
+                authors_list.extend([x.strip() for x in author.split(',')])
+            else:
+                authors_list.append(author.strip())
+    return authors_list
+
+
 def get_sorted_author(value):
    try:
        if ',' not in value:
@@ -270,7 +284,10 @@ def get_sorted_author(value):
            combined = "(" + ")|(".join(regexes) + ")"
            value = value.split(" ")
            if re.match(combined, value[-1].upper()):
-                value2 = value[-2] + ", " + " ".join(value[:-2]) + " " + value[-1]
+                if len(value) > 1:
+                    value2 = value[-2] + ", " + " ".join(value[:-2]) + " " + value[-1]
+                else:
+                    value2 = value[0]
            elif len(value) == 1:
                value2 = value[0]
            else:
@@ -279,7 +296,10 @@ def get_sorted_author(value):
            value2 = value
    except Exception as ex:
        log.error("Sorting author %s failed: %s", value, ex)
-        value2 = value
+        if isinstance(list, value2):
+            value2 = value[0]
+        else:
+            value2 = value
    return value2


@@ -295,15 +315,16 @@ def delete_book_file(book, calibrepath, book_format=None):
            return True, None
        else:
            if os.path.isdir(path):
-                if len(next(os.walk(path))[1]):
-                    log.error("Deleting book %s failed, path has subfolders: %s", book.id, book.path)
-                    return False , _("Deleting book %(id)s failed, path has subfolders: %(path)s",
-                                     id=book.id,
-                                     path=book.path)
                try:
-                    for root, __, files in os.walk(path):
+                    for root, folders, files in os.walk(path):
                        for f in files:
                            os.unlink(os.path.join(root, f))
+                        if len(folders):
+                            log.warning("Deleting book {} failed, path {} has subfolders: {}".format(book.id,
+                                        book.path, folders))
+                            return True, _("Deleting bookfolder for book %(id)s failed, path has subfolders: %(path)s",
+                                           id=book.id,
+                                           path=book.path)
                    shutil.rmtree(path)
                except (IOError, OSError) as e:
                    log.error("Deleting book %s failed: %s", book.id, e)
@@ -317,13 +338,13 @@ def delete_book_file(book, calibrepath, book_format=None):
                return True, None
            else:
                log.error("Deleting book %s failed, book path not valid: %s", book.id, book.path)
-                return False, _("Deleting book %(id)s failed, book path not valid: %(path)s",
-                                     id=book.id,
-                                     path=book.path)
+                return True, _("Deleting book %(id)s, book path not valid: %(path)s",
+                               id=book.id,
+                               path=book.path)


 def update_dir_structure_file(book_id, calibrepath, first_author):
-    localbook = db.session.query(db.Books).filter(db.Books.id == book_id).first()
+    localbook = calibre_db.get_book(book_id)
    path = os.path.join(calibrepath, localbook.path)

    authordir = localbook.path.split('/')[0]
@@ -339,13 +360,13 @@ def update_dir_structure_file(book_id, calibrepath, first_author):
        new_title_path = os.path.join(os.path.dirname(path), new_titledir)
        try:
            if not os.path.exists(new_title_path):
-                os.renames(path, new_title_path)
+                os.renames(os.path.normcase(path), os.path.normcase(new_title_path))
            else:
                log.info("Copying title: %s into existing: %s", path, new_title_path)
                for dir_name, __, file_list in os.walk(path):
                    for file in file_list:
-                        os.renames(os.path.join(dir_name, file),
-                                   os.path.join(new_title_path + dir_name[len(path):], file))
+                        os.renames(os.path.normcase(os.path.join(dir_name, file)),
+                                   os.path.normcase(os.path.join(new_title_path + dir_name[len(path):], file)))
            path = new_title_path
            localbook.path = localbook.path.split('/')[0] + '/' + new_titledir
        except OSError as ex:
@@ -356,7 +377,7 @@ def update_dir_structure_file(book_id, calibrepath, first_author):
    if authordir != new_authordir:
        new_author_path = os.path.join(calibrepath, new_authordir, os.path.basename(path))
        try:
-            os.renames(path, new_author_path)
+            os.renames(os.path.normcase(path), os.path.normcase(new_author_path))
            localbook.path = new_authordir + '/' + localbook.path.split('/')[1]
        except OSError as ex:
            log.error("Rename author from: %s to %s: %s", path, new_author_path, ex)
@@ -365,12 +386,14 @@ def update_dir_structure_file(book_id, calibrepath, first_author):
                     src=path, dest=new_author_path, error=str(ex))
    # Rename all files from old names to new names
    if authordir != new_authordir or titledir != new_titledir:
+        new_name = ""
        try:
            new_name = get_valid_filename(localbook.title) + ' - ' + get_valid_filename(new_authordir)
            path_name = os.path.join(calibrepath, new_authordir, os.path.basename(path))
            for file_format in localbook.data:
-                os.renames(os.path.join(path_name, file_format.name + '.' + file_format.format.lower()),
-                           os.path.join(path_name, new_name + '.' + file_format.format.lower()))
+                os.renames(os.path.normcase(
+                    os.path.join(path_name, file_format.name + '.' + file_format.format.lower())),
+                           os.path.normcase(os.path.join(path_name, new_name + '.' + file_format.format.lower())))
                file_format.name = new_name
        except OSError as ex:
            log.error("Rename file in path %s to %s: %s", path, new_name, ex)
@@ -382,7 +405,7 @@ def update_dir_structure_file(book_id, calibrepath, first_author):

 def update_dir_structure_gdrive(book_id, first_author):
    error = False
-    book = db.session.query(db.Books).filter(db.Books.id == book_id).first()
+    book = calibre_db.get_book(book_id)
    path = book.path

    authordir = book.path.split('/')[0]
@@ -450,11 +473,11 @@ def reset_password(user_id):
    existing_user = ub.session.query(ub.User).filter(ub.User.id == user_id).first()
    if not existing_user:
        return 0, None
-    password = generate_random_password()
-    existing_user.password = generate_password_hash(password)
    if not config.get_mail_server_configured():
        return 2, None
    try:
+        password = generate_random_password()
+        existing_user.password = generate_password_hash(password)
        ub.session.commit()
        send_registration_mail(existing_user.email, existing_user.nickname, password, True)
        return 1, existing_user.nickname
@@ -466,9 +489,20 @@ def reset_password(user_id):
 def generate_random_password():
    s = "abcdefghijklmnopqrstuvwxyz01234567890ABCDEFGHIJKLMNOPQRSTUVWXYZ!@#$%&*()?"
    passlen = 8
-    return "".join(random.sample(s, passlen))
+    if sys.version_info < (3, 0):
+        return "".join(s[ord(c) % len(s)] for c in os.urandom(passlen))
+    else:
+        return "".join(s[c % len(s)] for c in os.urandom(passlen))

-################################## External interface
+
+def uniq(inpt):
+    output = []
+    for x in inpt:
+        if x not in output:
+            output.append(x)
+    return output
+
+# ################################# External interface #################################


 def update_dir_stucture(book_id, calibrepath, first_author=None):
@@ -493,18 +527,17 @@ def get_cover_on_failure(use_generic_cover):


 def get_book_cover(book_id):
-    book = db.session.query(db.Books).filter(db.Books.id == book_id).filter(common_filters()).first()
+    book = calibre_db.get_filtered_book(book_id, allow_show_archived=True)
    return get_book_cover_internal(book, use_generic_cover_on_failure=True)


 def get_book_cover_with_uuid(book_uuid,
                             use_generic_cover_on_failure=True):
-    book = db.session.query(db.Books).filter(db.Books.uuid == book_uuid).first()
+    book = calibre_db.get_book_by_uuid(book_uuid)
    return get_book_cover_internal(book, use_generic_cover_on_failure)


-def get_book_cover_internal(book,
-                            use_generic_cover_on_failure):
+def get_book_cover_internal(book, use_generic_cover_on_failure):
    if book and book.has_cover:
        if config.config_use_google_drive:
            try:
@@ -532,8 +565,18 @@ def get_book_cover_internal(book,

 # saves book cover from url
 def save_cover_from_url(url, book_path):
-    img = requests.get(url, timeout=10)      # ToDo: Error Handling
-    return save_cover(img, book_path)
+    try:
+        img = requests.get(url, timeout=(10, 200))      # ToDo: Error Handling
+        img.raise_for_status()
+        return save_cover(img, book_path)
+    except (requests.exceptions.HTTPError,
+            requests.exceptions.ConnectionError,
+            requests.exceptions.Timeout) as ex:
+        log.info(u'Cover Download Error %s', ex)
+        return False, _("Error Downloading Cover")
+    except UnidentifiedImageError as ex:
+        log.info(u'File Format Error %s', ex)
+        return False, _("Cover Format Error")


 def save_cover_from_filestorage(filepath, saved_filename, img):
@@ -594,7 +637,7 @@ def save_cover(img, book_path):
        return save_cover_from_filestorage(os.path.join(config.config_calibre_dir, book_path), "cover.jpg", img)


-def do_download_file(book, book_format, data, headers):
+def do_download_file(book, book_format, client, data, headers):
    if config.config_use_google_drive:
        startTime = time.time()
        df = gd.getFileFromEbooksFolder(book.path, data.name + "." + book_format)
@@ -608,8 +651,14 @@ def do_download_file(book, book_format, data, headers):
        if not os.path.isfile(os.path.join(filename, data.name + "." + book_format)):
            # ToDo: improve error handling
            log.error('File not found: %s', os.path.join(filename, data.name + "." + book_format))
+
+        if client == "kobo" and book_format == "kepub":
+            headers["Content-Disposition"] = headers["Content-Disposition"].replace(".kepub", ".kepub.epub")
+
        response = make_response(send_from_directory(filename, data.name + "." + book_format))
-        response.headers = headers
+        # ToDo Check headers parameter
+        for element in headers:
+            response.headers[element[0]] = element[1]
        return response

 ##################################
@@ -627,11 +676,12 @@ def check_unrar(unrarLocation):
            unrarLocation = unrarLocation.encode(sys.getfilesystemencoding())
        unrarLocation = [unrarLocation]
        for lines in process_wait(unrarLocation):
-            value = re.search('UNRAR (.*) freeware', lines)
+            value = re.search('UNRAR (.*) freeware', lines, re.IGNORECASE)
            if value:
                version = value.group(1)
                log.debug("unrar version %s", version)
-    except OSError as err:
+                break
+    except (OSError, UnicodeDecodeError) as err:
        log.exception(err)
        return _('Error excecuting UnRar')

@@ -687,7 +737,7 @@ def render_task_status(tasklist):
                task['runtime'] = format_runtime(task['formRuntime'])

            # localize the task status
-            if isinstance( task['stat'], int):
+            if isinstance(task['stat'], int):
                if task['stat'] == STAT_WAITING:
                    task['status'] = _(u'Waiting')
                elif task['stat'] == STAT_FAIL:
@@ -700,7 +750,7 @@ def render_task_status(tasklist):
                    task['status'] = _(u'Unknown Status')

            # localize the task type
-            if isinstance( task['taskType'], int):
+            if isinstance(task['taskType'], int):
                if task['taskType'] == TASK_EMAIL:
                    task['taskMessage'] = _(u'E-mail: ') + task['taskMess']
                elif task['taskType'] == TASK_CONVERT:
@@ -717,66 +767,12 @@ def render_task_status(tasklist):
    return renderedtasklist


-# Language and content filters for displaying in the UI
-def common_filters(allow_show_archived=False):
-    if not allow_show_archived:
-        archived_books = (
-            ub.session.query(ub.ArchivedBook)
-            .filter(ub.ArchivedBook.user_id == int(current_user.id))
-            .filter(ub.ArchivedBook.is_archived == True)
-            .all()
-        )
-        archived_book_ids = [archived_book.book_id for archived_book in archived_books]
-        archived_filter = db.Books.id.notin_(archived_book_ids)
-    else:
-        archived_filter = true()
-
-    if current_user.filter_language() != "all":
-        lang_filter = db.Books.languages.any(db.Languages.lang_code == current_user.filter_language())
-    else:
-        lang_filter = true()
-    negtags_list = current_user.list_denied_tags()
-    postags_list = current_user.list_allowed_tags()
-    neg_content_tags_filter = false() if negtags_list == [''] else db.Books.tags.any(db.Tags.name.in_(negtags_list))
-    pos_content_tags_filter = true() if postags_list == [''] else db.Books.tags.any(db.Tags.name.in_(postags_list))
-    if config.config_restricted_column:
-        pos_cc_list = current_user.allowed_column_value.split(',')
-        pos_content_cc_filter = true() if pos_cc_list == [''] else \
-            getattr(db.Books, 'custom_column_' + str(config.config_restricted_column)).\
-                    any(db.cc_classes[config.config_restricted_column].value.in_(pos_cc_list))
-        neg_cc_list = current_user.denied_column_value.split(',')
-        neg_content_cc_filter = false() if neg_cc_list == [''] else \
-            getattr(db.Books, 'custom_column_' + str(config.config_restricted_column)).\
-                    any(db.cc_classes[config.config_restricted_column].value.in_(neg_cc_list))
-    else:
-        pos_content_cc_filter = true()
-        neg_content_cc_filter = false()
-    return and_(lang_filter, pos_content_tags_filter, ~neg_content_tags_filter,
-                pos_content_cc_filter, ~neg_content_cc_filter, archived_filter)
-
-
 def tags_filters():
    negtags_list = current_user.list_denied_tags()
    postags_list = current_user.list_allowed_tags()
    neg_content_tags_filter = false() if negtags_list == [''] else db.Tags.name.in_(negtags_list)
    pos_content_tags_filter = true() if postags_list == [''] else db.Tags.name.in_(postags_list)
    return and_(pos_content_tags_filter, ~neg_content_tags_filter)
-    # return ~(false()) if postags_list == [''] else db.Tags.in_(postags_list)
-
-
-# Creates for all stored languages a translated speaking name in the array for the UI
-def speaking_language(languages=None):
-    if not languages:
-        languages = db.session.query(db.Languages).join(db.books_languages_link).join(db.Books)\
-            .filter(common_filters())\
-            .group_by(text('books_languages_link.lang_code')).all()
-    for lang in languages:
-        try:
-            cur_l = LC.parse(lang.lang_code)
-            lang.name = cur_l.get_language_name(get_locale())
-        except UnknownLocaleError:
-            lang.name = _(isoLanguages.get(part3=lang.lang_code).name)
-    return languages


 # checks if domain is in database (including wildcards)
@@ -793,93 +789,29 @@ def check_valid_domain(domain_text):
    return not len(result)


-# Orders all Authors in the list according to authors sort
-def order_authors(entry):
-    sort_authors = entry.author_sort.split('&')
-    authors_ordered = list()
-    error = False
-    for auth in sort_authors:
-        # ToDo: How to handle not found authorname
-        result = db.session.query(db.Authors).filter(db.Authors.sort == auth.lstrip().strip()).first()
-        if not result:
-            error = True
-            break
-        authors_ordered.append(result)
-    if not error:
-        entry.authors = authors_ordered
-    return entry
-
-
-# Fill indexpage with all requested data from database
-def fill_indexpage(page, database, db_filter, order, *join):
-    return fill_indexpage_with_archived_books(page, database, db_filter, order, False, *join)
-
-
-def fill_indexpage_with_archived_books(page, database, db_filter, order, allow_show_archived, *join):
-    if current_user.show_detail_random():
-        randm = db.session.query(db.Books).filter(common_filters(allow_show_archived))\
-            .order_by(func.random()).limit(config.config_random_books)
-    else:
-        randm = false()
-    off = int(int(config.config_books_per_page) * (page - 1))
-    pagination = Pagination(page, config.config_books_per_page,
-                            len(db.session.query(database).filter(db_filter)
-                                .filter(common_filters(allow_show_archived)).all()))
-    entries = db.session.query(database).join(*join, isouter=True).filter(db_filter)\
-        .filter(common_filters(allow_show_archived))\
-        .order_by(*order).offset(off).limit(config.config_books_per_page).all()
-    for book in entries:
-        book = order_authors(book)
-    return entries, randm, pagination
-
-
-def get_typeahead(database, query, replace=('', ''), tag_filter=true()):
-    query = query or ''
-    db.session.connection().connection.connection.create_function("lower", 1, lcase)
-    entries = db.session.query(database).filter(tag_filter).\
-        filter(func.lower(database.name).ilike("%" + query + "%")).all()
-    json_dumps = json.dumps([dict(name=r.name.replace(*replace)) for r in entries])
-    return json_dumps
-
-
-# read search results from calibre-database and return it (function is used for feed and simple search
-def get_search_results(term):
-    db.session.connection().connection.connection.create_function("lower", 1, lcase)
-    q = list()
-    authorterms = re.split("[, ]+", term)
-    for authorterm in authorterms:
-        q.append(db.Books.authors.any(func.lower(db.Authors.name).ilike("%" + authorterm + "%")))
-
-    db.Books.authors.any(func.lower(db.Authors.name).ilike("%" + term + "%"))
-
-    return db.session.query(db.Books).filter(common_filters()).filter(
-        or_(db.Books.tags.any(func.lower(db.Tags.name).ilike("%" + term + "%")),
-            db.Books.series.any(func.lower(db.Series.name).ilike("%" + term + "%")),
-            db.Books.authors.any(and_(*q)),
-            db.Books.publishers.any(func.lower(db.Publishers.name).ilike("%" + term + "%")),
-            func.lower(db.Books.title).ilike("%" + term + "%")
-            )).order_by(db.Books.sort).all()
-
-
-def get_cc_columns():
-    tmpcc = db.session.query(db.Custom_Columns).filter(db.Custom_Columns.datatype.notin_(db.cc_exceptions)).all()
+def get_cc_columns(filter_config_custom_read=False):
+    tmpcc = calibre_db.session.query(db.Custom_Columns)\
+        .filter(db.Custom_Columns.datatype.notin_(db.cc_exceptions)).all()
+    cc = []
+    r = None
    if config.config_columns_to_ignore:
-        cc = []
-        for col in tmpcc:
-            r = re.compile(config.config_columns_to_ignore)
-            if not r.match(col.name):
-                cc.append(col)
-    else:
-        cc = tmpcc
+        r = re.compile(config.config_columns_to_ignore)
+
+    for col in tmpcc:
+        if filter_config_custom_read and config.config_read_column and config.config_read_column == col.id:
+            continue
+        if r and r.match(col.name):
+            continue
+        cc.append(col)
+
    return cc


-def get_download_link(book_id, book_format):
+def get_download_link(book_id, book_format, client):
    book_format = book_format.split(".")[0]
-    book = db.session.query(db.Books).filter(db.Books.id == book_id).filter(common_filters()).first()
+    book = calibre_db.get_filtered_book(book_id)
    if book:
-        data1 = db.session.query(db.Data).filter(db.Data.book == book.id)\
-            .filter(db.Data.format == book_format.upper()).first()
+        data1 = calibre_db.get_book_format(book.id, book_format.upper())
    else:
        abort(404)
    if data1:
@@ -894,28 +826,6 @@ def get_download_link(book_id, book_format):
        headers["Content-Type"] = mimetypes.types_map.get('.' + book_format, "application/octet-stream")
        headers["Content-Disposition"] = "attachment; filename=%s.%s; filename*=UTF-8''%s.%s" % (
            quote(file_name.encode('utf-8')), book_format, quote(file_name.encode('utf-8')), book_format)
-        return do_download_file(book, book_format, data1, headers)
+        return do_download_file(book, book_format, client, data1, headers)
    else:
        abort(404)
-
-
-def check_exists_book(authr, title):
-    db.session.connection().connection.connection.create_function("lower", 1, lcase)
-    q = list()
-    authorterms = re.split(r'\s*&\s*', authr)
-    for authorterm in authorterms:
-        q.append(db.Books.authors.any(func.lower(db.Authors.name).ilike("%" + authorterm + "%")))
-
-    return db.session.query(db.Books).filter(
-        and_(db.Books.authors.any(and_(*q)),
-             func.lower(db.Books.title).ilike("%" + title + "%")
-             )).first()
-
-############### Database Helper functions
-
-
-def lcase(s):
-    try:
-        return unidecode.unidecode(s.lower())
-    except Exception as e:
-        log.exception(e)
@@ -57,12 +57,36 @@ def get_language_name(locale, lang_code):

 def get_language_codes(locale, language_names, remainder=None):
    language_names = set(x.strip().lower() for x in language_names if x)
-
+    languages = list()
    for k, v in get_language_names(locale).items():
        v = v.lower()
        if v in language_names:
+            languages.append(k)
            language_names.remove(v)
-            yield k
-
    if remainder is not None:
        remainder.extend(language_names)
+    return languages
+
+def get_valid_language_codes(locale, language_names, remainder=None):
+    languages = list()
+    if "" in language_names:
+        language_names.remove("")
+    for k, v in get_language_names(locale).items():
+        if k in language_names:
+            languages.append(k)
+            language_names.remove(k)
+    if remainder is not None and len(language_names):
+        remainder.extend(language_names)
+    return languages
+
+def get_lang3(lang):
+    try:
+        if len(lang) == 2:
+            ret_value = get(part1=lang).part3
+        elif len(lang) == 3:
+            ret_value = lang
+        else:
+            ret_value = ""
+    except KeyError:
+        ret_value = lang
+    return ret_value
@@ -112,9 +112,9 @@ def timestamptodate(date, fmt=None):
 def yesno(value, yes, no):
    return yes if value else no

-
-'''@jinjia.app_template_filter('canread')
-def canread(ext):
-    if isinstance(ext, db.Data):
-        ext = ext.format
-    return ext.lower() in EXTENSIONS_READER'''
+@jinjia.app_template_filter('formatfloat')
+def formatfloat(value, decimals=1):
+    formatedstring = '%d' % value
+    if (value % 1) != 0:
+        formatedstring = ('%s.%d' % (formatedstring, (value % 1) * 10**decimals)).rstrip('0')
+    return formatedstring
@@ -19,8 +19,6 @@

 import base64
 import datetime
-import itertools
-import json
 import sys
 import os
 import uuid
@@ -48,7 +46,7 @@ from sqlalchemy.sql.expression import and_, or_
 from sqlalchemy.exc import StatementError
 import requests

-from . import config, logger, kobo_auth, db, helper, shelf as shelf_lib, ub
+from . import config, logger, kobo_auth, db, calibre_db, helper, shelf as shelf_lib, ub
 from .services import SyncToken as SyncToken
 from .web import download_required
 from .kobo_auth import requires_kobo_auth
@@ -131,7 +129,7 @@ def HandleSyncRequest():
    sync_token = SyncToken.SyncToken.from_headers(request.headers)
    log.info("Kobo library sync request received.")
    if not current_app.wsgi_app.is_proxied:
-        log.debug('Kobo: Received unproxied request, changed request port to server port')
+        log.debug('Kobo: Received unproxied request, changed request port to external server port')

    # TODO: Limit the number of books return per sync call, and rely on the sync-continuatation header
    # instead so that the device triggers another sync.
@@ -143,7 +141,7 @@ def HandleSyncRequest():

    # We reload the book database so that the user get's a fresh view of the library
    # in case of external changes (e.g: adding a book through Calibre).
-    db.reconnect_db(config)
+    calibre_db.reconnect_db(config, ub.app_DB_path)

    archived_books = (
        ub.session.query(ub.ArchivedBook)
@@ -170,7 +168,7 @@ def HandleSyncRequest():
    # It looks like it's treating the db.Books.last_modified field as a string and may fail
    # the comparison because of the +00:00 suffix.
    changed_entries = (
-        db.session.query(db.Books)
+        calibre_db.session.query(db.Books)
        .join(db.Data)
        .filter(or_(func.datetime(db.Books.last_modified) > sync_token.books_last_modified,
                    db.Books.id.in_(recently_restored_or_archived_books)))
@@ -207,7 +205,7 @@ def HandleSyncRequest():
                     ub.KoboReadingState.user_id == current_user.id,
                     ub.KoboReadingState.book_id.notin_(reading_states_in_new_entitlements))))
    for kobo_reading_state in changed_reading_states.all():
-        book = db.session.query(db.Books).filter(db.Books.id == kobo_reading_state.book_id).one_or_none()
+        book = calibre_db.session.query(db.Books).filter(db.Books.id == kobo_reading_state.book_id).one_or_none()
        if book:
            sync_results.append({
                "ChangedReadingState": {
@@ -254,9 +252,9 @@ def generate_sync_response(sync_token, sync_results):
@download_required
 def HandleMetadataRequest(book_uuid):
    if not current_app.wsgi_app.is_proxied:
-        log.debug('Kobo: Received unproxied request, changed request port to server port')
+        log.debug('Kobo: Received unproxied request, changed request port to external server port')
    log.info("Kobo library metadata request received for book %s" % book_uuid)
-    book = db.session.query(db.Books).filter(db.Books.uuid == book_uuid).first()
+    book = calibre_db.get_book_by_uuid(book_uuid)
    if not book or not book.data:
        log.info(u"Book %s not found in database", book_uuid)
        return redirect_or_proxy_request()
@@ -267,14 +265,15 @@ def HandleMetadataRequest(book_uuid):

 def get_download_url_for_book(book, book_format):
    if not current_app.wsgi_app.is_proxied:
-        if ':' in request.host and not request.host.endswith(']') :
+        if ':' in request.host and not request.host.endswith(']'):
            host = "".join(request.host.split(':')[:-1])
        else:
            host = request.host
+
        return "{url_scheme}://{url_base}:{url_port}/download/{book_id}/{book_format}".format(
            url_scheme=request.scheme,
            url_base=host,
-            url_port=config.config_port,
+            url_port=config.config_external_port,
            book_id=book.id,
            book_format=book_format.lower()
        )
@@ -317,8 +316,15 @@ def get_description(book):
 # TODO handle multiple authors
 def get_author(book):
    if not book.authors:
-        return None
-    return book.authors[0].name
+        return {"Contributors": None}
+    if len(book.authors) > 1:
+        author_list = []
+        autor_roles = []
+        for author in book.authors:
+            autor_roles.append({"Name":author.name, "Role":"Author"})
+            author_list.append(author.name)
+        return {"ContributorRoles": autor_roles, "Contributors":author_list}
+    return {"ContributorRoles": [{"Name":book.authors[0].name, "Role":"Author"}], "Contributors": book.authors[0].name}


 def get_publisher(book):
@@ -332,6 +338,9 @@ def get_series(book):
        return None
    return book.series[0].name

+def get_seriesindex(book):
+    return book.series_index or 1
+

 def get_metadata(book):
    download_urls = []
@@ -354,7 +363,7 @@ def get_metadata(book):
    book_uuid = book.uuid
    metadata = {
        "Categories": ["00000000-0000-0000-0000-000000000001",],
-        "Contributors": get_author(book),
+        # "Contributors": get_author(book),
        "CoverImageId": book_uuid,
        "CrossRevisionId": book_uuid,
        "CurrentDisplayPrice": {"CurrencyCode": "USD", "TotalAmount": 0},
@@ -378,6 +387,7 @@ def get_metadata(book):
        "Title": book.title,
        "WorkId": book_uuid,
    }
+    metadata.update(get_author(book))

    if get_series(book):
        if sys.version_info < (3, 0):
@@ -386,8 +396,8 @@ def get_metadata(book):
            name = get_series(book)
        metadata["Series"] = {
            "Name": get_series(book),
-            "Number": book.series_index,        # ToDo Check int() ?
-            "NumberFloat": float(book.series_index),
+            "Number": get_seriesindex(book),        # ToDo Check int() ?
+            "NumberFloat": float(get_seriesindex(book)),
            # Get a deterministic id based on the series name.
            "Id": uuid.uuid3(uuid.NAMESPACE_DNS, name),
        }
@@ -396,7 +406,7 @@ def get_metadata(book):


@kobo.route("/v1/library/tags", methods=["POST", "DELETE"])
-@login_required
+@requires_kobo_auth
 # Creates a Shelf with the given items, and returns the shelf's uuid.
 def HandleTagCreate():
    # catch delete requests, otherwise the are handeld in the book delete handler
@@ -431,6 +441,7 @@ def HandleTagCreate():


@kobo.route("/v1/library/tags/<tag_id>", methods=["DELETE", "PUT"])
+@requires_kobo_auth
 def HandleTagUpdate(tag_id):
    shelf = ub.session.query(ub.Shelf).filter(ub.Shelf.uuid == tag_id,
                                              ub.Shelf.user_id == current_user.id).one_or_none()
@@ -471,7 +482,7 @@ def add_items_to_shelf(items, shelf):
                items_unknown_to_calibre.append(item)
                continue

-            book = db.session.query(db.Books).filter(db.Books.uuid == item["RevisionId"]).one_or_none()
+            book = calibre_db.get_book_by_uuid(item["RevisionId"])
            if not book:
                items_unknown_to_calibre.append(item)
                continue
@@ -485,7 +496,7 @@ def add_items_to_shelf(items, shelf):


@kobo.route("/v1/library/tags/<tag_id>/items", methods=["POST"])
-@login_required
+@requires_kobo_auth
 def HandleTagAddItem(tag_id):
    items = None
    try:
@@ -515,7 +526,7 @@ def HandleTagAddItem(tag_id):


@kobo.route("/v1/library/tags/<tag_id>/items/delete", methods=["POST"])
-@login_required
+@requires_kobo_auth
 def HandleTagRemoveItem(tag_id):
    items = None
    try:
@@ -542,7 +553,7 @@ def HandleTagRemoveItem(tag_id):
                items_unknown_to_calibre.append(item)
                continue

-            book = db.session.query(db.Books).filter(db.Books.uuid == item["RevisionId"]).one_or_none()
+            book = calibre_db.get_book_by_uuid(item["RevisionId"])
            if not book:
                items_unknown_to_calibre.append(item)
                continue
@@ -610,7 +621,7 @@ def create_kobo_tag(shelf):
        "Type": "UserTag"
    }
    for book_shelf in shelf.books:
-        book = db.session.query(db.Books).filter(db.Books.id == book_shelf.book_id).one_or_none()
+        book = calibre_db.get_book(book_shelf.book_id)
        if not book:
            log.info(u"Book (id: %s) in BookShelf (id: %s) not found in book database",  book_shelf.book_id, shelf.id)
            continue
@@ -624,9 +635,9 @@ def create_kobo_tag(shelf):


@kobo.route("/v1/library/<book_uuid>/state", methods=["GET", "PUT"])
-@login_required
+@requires_kobo_auth
 def HandleStateRequest(book_uuid):
-    book = db.session.query(db.Books).filter(db.Books.uuid == book_uuid).first()
+    book = calibre_db.get_book_by_uuid(book_uuid)
    if not book or not book.data:
        log.info(u"Book %s not found in database", book_uuid)
        return redirect_or_proxy_request()
@@ -798,10 +809,10 @@ def TopLevelEndpoint():


@kobo.route("/v1/library/<book_uuid>", methods=["DELETE"])
-@login_required
+@requires_kobo_auth
 def HandleBookDeletionRequest(book_uuid):
    log.info("Kobo book deletion request received for book %s" % book_uuid)
-    book = db.session.query(db.Books).filter(db.Books.uuid == book_uuid).first()
+    book = calibre_db.get_book_by_uuid(book_uuid)
    if not book:
        log.info(u"Book %s not found in database", book_uuid)
        return redirect_or_proxy_request()
@@ -914,7 +925,7 @@ def HandleInitRequest():
        kobo_resources = NATIVE_KOBO_RESOURCES()

    if not current_app.wsgi_app.is_proxied:
-        log.debug('Kobo: Received unproxied request, changed request port to server port')
+        log.debug('Kobo: Received unproxied request, changed request port to external server port')
        if ':' in request.host and not request.host.endswith(']'):
            host = "".join(request.host.split(':')[:-1])
        else:
@@ -922,8 +933,9 @@ def HandleInitRequest():
        calibre_web_url = "{url_scheme}://{url_base}:{url_port}".format(
            url_scheme=request.scheme,
            url_base=host,
-            url_port=config.config_port
+            url_port=config.config_external_port
        )
+        log.debug('Kobo: Received unproxied request, changed request url to %s', calibre_web_url)
        kobo_resources["image_host"] = calibre_web_url
        kobo_resources["image_url_quality_template"] = unquote(calibre_web_url +
                                                               url_for("kobo.HandleCoverImageRequest",
@@ -932,16 +944,14 @@ def HandleInitRequest():
                                                                       width="{width}",
                                                                       height="{height}",
                                                                       Quality='{Quality}',
-                                                                       isGreyscale='isGreyscale'
-                                                               ))
+                                                                       isGreyscale='isGreyscale'))
        kobo_resources["image_url_template"] = unquote(calibre_web_url +
                                                       url_for("kobo.HandleCoverImageRequest",
                                                               auth_token=kobo_auth.get_auth_token(),
                                                               book_uuid="{ImageId}",
                                                               width="{width}",
                                                               height="{height}",
-                                                               isGreyscale='false'
-                                                       ))
+                                                               isGreyscale='false'))
    else:
        kobo_resources["image_host"] = url_for("web.index", _external=True).strip("/")
        kobo_resources["image_url_quality_template"] = unquote(url_for("kobo.HandleCoverImageRequest",
@@ -960,7 +970,6 @@ def HandleInitRequest():
                                                               isGreyscale='false',
                                                               _external=True))

-
    response = make_response(jsonify({"Resources": kobo_resources}))
    response.headers["x-kobo-apitoken"] = "e30="

@@ -979,7 +988,7 @@ def NATIVE_KOBO_RESOURCES():
        "blackstone_header": {"key": "x-amz-request-payer", "value": "requester"},
        "book": "https://storeapi.kobo.com/v1/products/books/{ProductId}",
        "book_detail_page": "https://store.kobobooks.com/{culture}/ebook/{slug}",
-        "book_detail_page_rakuten": "http://books.rakuten.co.jp/rk/{crossrevisionid}",
+        "book_detail_page_rakuten": "https://books.rakuten.co.jp/rk/{crossrevisionid}",
        "book_landing_page": "https://store.kobobooks.com/ebooks",
        "book_subscription": "https://storeapi.kobo.com/v1/products/books/subscriptions",
        "categories": "https://storeapi.kobo.com/v1/categories",
@@ -1017,7 +1026,7 @@ def NATIVE_KOBO_RESOURCES():
        "get_tests_request": "https://storeapi.kobo.com/v1/analytics/gettests",
        "giftcard_epd_redeem_url": "https://www.kobo.com/{storefront}/{language}/redeem-ereader",
        "giftcard_redeem_url": "https://www.kobo.com/{storefront}/{language}/redeem",
-        "help_page": "http://www.kobo.com/help",
+        "help_page": "https://www.kobo.com/help",
        "kobo_audiobooks_enabled": "False",
        "kobo_audiobooks_orange_deal_enabled": "False",
        "kobo_audiobooks_subscriptions_enabled": "False",
@@ -121,7 +121,11 @@ kobo_auth = Blueprint("kobo_auth", __name__, url_prefix="/kobo_auth")
@kobo_auth.route("/generate_auth_token/<int:user_id>")
@login_required
 def generate_auth_token(user_id):
-    host = ':'.join(request.host.rsplit(':')[0:-1])
+    host_list = request.host.rsplit(':')
+    if len(host_list) == 1:
+        host = ':'.join(host_list)
+    else:
+        host = ':'.join(host_list[0:-1])
    if host.startswith('127.') or host.lower() == 'localhost' or host.startswith('[::ffff:7f'):
        warning = _('PLease access calibre-web from non localhost to get valid api_endpoint for kobo device')
        return render_title_template(
@@ -82,7 +82,6 @@ def _absolute_log_file(log_file, default_log_file):
        if not os.path.dirname(log_file):
            log_file = os.path.join(_CONFIG_DIR, log_file)
        return os.path.abspath(log_file)
-
    return default_log_file


@@ -115,7 +114,7 @@ def setup(log_file, log_level=None):
    if previous_handler:
        # if the log_file has not changed, don't create a new handler
        if getattr(previous_handler, 'baseFilename', None) == log_file:
-            return
+            return "" if log_file == DEFAULT_LOG_FILE else log_file
        logging.debug("logging to %s level %s", log_file, r.level)

    if log_file == LOG_TO_STDERR or log_file == LOG_TO_STDOUT:
@@ -127,17 +126,19 @@ def setup(log_file, log_level=None):
            file_handler.baseFilename = log_file
    else:
        try:
-            file_handler = RotatingFileHandler(log_file, maxBytes=50000, backupCount=2)
+            file_handler = RotatingFileHandler(log_file, maxBytes=50000, backupCount=2, encoding='utf-8')
        except IOError:
            if log_file == DEFAULT_LOG_FILE:
                raise
-            file_handler = RotatingFileHandler(DEFAULT_LOG_FILE, maxBytes=50000, backupCount=2)
+            file_handler = RotatingFileHandler(DEFAULT_LOG_FILE, maxBytes=50000, backupCount=2, encoding='utf-8')
+            log_file = ""
    file_handler.setFormatter(FORMATTER)

    for h in r.handlers:
        r.removeHandler(h)
        h.close()
    r.addHandler(file_handler)
+    return "" if log_file == DEFAULT_LOG_FILE else log_file


 def create_access_log(log_file, log_name, formatter):
@@ -150,11 +151,18 @@ def create_access_log(log_file, log_name, formatter):
    access_log = logging.getLogger(log_name)
    access_log.propagate = False
    access_log.setLevel(logging.INFO)
+    try:
+        file_handler = RotatingFileHandler(log_file, maxBytes=50000, backupCount=2, encoding='utf-8')
+    except IOError:
+        if log_file == DEFAULT_ACCESS_LOG:
+            raise
+        file_handler = RotatingFileHandler(DEFAULT_ACCESS_LOG, maxBytes=50000, backupCount=2, encoding='utf-8')
+        log_file = ""

-    file_handler = RotatingFileHandler(log_file, maxBytes=50000, backupCount=2)
    file_handler.setFormatter(formatter)
    access_log.addHandler(file_handler)
-    return access_log
+    return access_log, \
+           "" if _absolute_log_file(log_file, DEFAULT_ACCESS_LOG) == DEFAULT_ACCESS_LOG else log_file


 # Enable logging of smtp lib debug output
@@ -40,7 +40,7 @@ try:
        Stores and retrieves OAuth tokens using a relational database through
        the `SQLAlchemy`_ ORM.

-        .. _SQLAlchemy: http://www.sqlalchemy.org/
+        .. _SQLAlchemy: https://www.sqlalchemy.org/
        """
        def __init__(self, model, session, provider_id,
                     user=None, user_id=None, user_required=None, anon_user=None,
@@ -122,10 +122,10 @@ if ub.oauth_support:
    ele2 = dict(provider_name='google',
                id=oauth_ids[1].id,
                active=oauth_ids[1].active,
-                scope=["https://www.googleapis.com/auth/plus.me", "https://www.googleapis.com/auth/userinfo.email"],
+                scope=["https://www.googleapis.com/auth/userinfo.email"],
                oauth_client_id=oauth_ids[1].oauth_client_id,
                oauth_client_secret=oauth_ids[1].oauth_client_secret,
-                obtain_link='https://github.com/settings/developers')
+                obtain_link='https://console.developers.google.com/apis/credentials')
    oauthblueprints.append(ele1)
    oauthblueprints.append(ele2)

@@ -287,7 +287,7 @@ if ub.oauth_support:
                    flash(_(u"Unlink to %(oauth)s Failed", oauth=oauth_check[provider]), category="error")
        except NoResultFound:
            log.warning("oauth %s for user %d not found", provider, current_user.id)
-            flash(_(u"Not Linked to %(oauth)s.", oauth=oauth_check[provider]), category="error")
+            flash(_(u"Not Linked to %(oauth)s", oauth=provider), category="error")
        return redirect(url_for('web.profile'))


@@ -355,4 +355,4 @@ if ub.oauth_support:
    @oauth.route('/unlink/google', methods=["GET"])
    @login_required
    def google_login_unlink():
-        return unlink_oauth(oauthblueprints[1]['blueprint'].name)
+        return unlink_oauth(oauthblueprints[1]['id'])
@@ -30,10 +30,10 @@ from flask_login import current_user
 from sqlalchemy.sql.expression import func, text, or_, and_
 from werkzeug.security import check_password_hash

-from . import constants, logger, config, db, ub, services, get_locale, isoLanguages
-from .helper import fill_indexpage, get_download_link, get_book_cover, speaking_language
+from . import constants, logger, config, db, calibre_db, ub, services, get_locale, isoLanguages
+from .helper import get_download_link, get_book_cover
 from .pagination import Pagination
-from .web import common_filters, get_search_results, render_read_books, download_required
+from .web import render_read_books, download_required
 from flask_babel import gettext as _
 from babel import Locale as LC
 from babel.core import UnknownLocaleError
@@ -100,15 +100,15 @@ def feed_normal_search():
@requires_basic_auth_if_no_ano
 def feed_new():
    off = request.args.get("offset") or 0
-    entries, __, pagination = fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
-                                             db.Books, True, [db.Books.timestamp.desc()])
+    entries, __, pagination = calibre_db.fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
+                                                        db.Books, True, [db.Books.timestamp.desc()])
    return render_xml_template('feed.xml', entries=entries, pagination=pagination)


@opds.route("/opds/discover")
@requires_basic_auth_if_no_ano
 def feed_discover():
-    entries = db.session.query(db.Books).filter(common_filters()).order_by(func.random())\
+    entries = calibre_db.session.query(db.Books).filter(calibre_db.common_filters()).order_by(func.random())\
        .limit(config.config_books_per_page)
    pagination = Pagination(1, config.config_books_per_page, int(config.config_books_per_page))
    return render_xml_template('feed.xml', entries=entries, pagination=pagination)
@@ -118,9 +118,9 @@ def feed_discover():
@requires_basic_auth_if_no_ano
 def feed_best_rated():
    off = request.args.get("offset") or 0
-    entries, __, pagination = fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
-                                             db.Books, db.Books.ratings.any(db.Ratings.rating > 9),
-                                             [db.Books.timestamp.desc()])
+    entries, __, pagination = calibre_db.fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
+                                                        db.Books, db.Books.ratings.any(db.Ratings.rating > 9),
+                                                        [db.Books.timestamp.desc()])
    return render_xml_template('feed.xml', entries=entries, pagination=pagination)


@@ -133,16 +133,13 @@ def feed_hot():
    hot_books = all_books.offset(off).limit(config.config_books_per_page)
    entries = list()
    for book in hot_books:
-        downloadBook = db.session.query(db.Books).filter(db.Books.id == book.Downloads.book_id).first()
+        downloadBook = calibre_db.get_book(book.Downloads.book_id)
        if downloadBook:
            entries.append(
-                db.session.query(db.Books).filter(common_filters())
-                .filter(db.Books.id == book.Downloads.book_id).first()
+                calibre_db.get_filtered_book(book.Downloads.book_id)
            )
        else:
            ub.delete_download(book.Downloads.book_id)
-            # ub.session.query(ub.Downloads).filter(book.Downloads.book_id == ub.Downloads.book_id).delete()
-            # ub.session.commit()
    numBooks = entries.__len__()
    pagination = Pagination((int(off) / (int(config.config_books_per_page)) + 1),
                            config.config_books_per_page, numBooks)
@@ -153,11 +150,13 @@ def feed_hot():
@requires_basic_auth_if_no_ano
 def feed_authorindex():
    off = request.args.get("offset") or 0
-    entries = db.session.query(db.Authors).join(db.books_authors_link).join(db.Books).filter(common_filters())\
-        .group_by(text('books_authors_link.author')).order_by(db.Authors.sort).limit(config.config_books_per_page)\
+    entries = calibre_db.session.query(db.Authors).join(db.books_authors_link).join(db.Books)\
+        .filter(calibre_db.common_filters())\
+        .group_by(text('books_authors_link.author'))\
+        .order_by(db.Authors.sort).limit(config.config_books_per_page)\
        .offset(off)
    pagination = Pagination((int(off) / (int(config.config_books_per_page)) + 1), config.config_books_per_page,
-                            len(db.session.query(db.Authors).all()))
+                            len(calibre_db.session.query(db.Authors).all()))
    return render_xml_template('feed.xml', listelements=entries, folder='opds.feed_author', pagination=pagination)


@@ -165,10 +164,10 @@ def feed_authorindex():
@requires_basic_auth_if_no_ano
 def feed_author(book_id):
    off = request.args.get("offset") or 0
-    entries, __, pagination = fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
-                                             db.Books,
-                                             db.Books.authors.any(db.Authors.id == book_id),
-                                             [db.Books.timestamp.desc()])
+    entries, __, pagination = calibre_db.fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
+                                                        db.Books,
+                                                        db.Books.authors.any(db.Authors.id == book_id),
+                                                        [db.Books.timestamp.desc()])
    return render_xml_template('feed.xml', entries=entries, pagination=pagination)


@@ -176,11 +175,14 @@ def feed_author(book_id):
@requires_basic_auth_if_no_ano
 def feed_publisherindex():
    off = request.args.get("offset") or 0
-    entries = db.session.query(db.Publishers).join(db.books_publishers_link).join(db.Books).filter(common_filters())\
-        .group_by(text('books_publishers_link.publisher')).order_by(db.Publishers.sort)\
+    entries = calibre_db.session.query(db.Publishers)\
+        .join(db.books_publishers_link)\
+        .join(db.Books).filter(calibre_db.common_filters())\
+        .group_by(text('books_publishers_link.publisher'))\
+        .order_by(db.Publishers.sort)\
        .limit(config.config_books_per_page).offset(off)
    pagination = Pagination((int(off) / (int(config.config_books_per_page)) + 1), config.config_books_per_page,
-                            len(db.session.query(db.Publishers).all()))
+                            len(calibre_db.session.query(db.Publishers).all()))
    return render_xml_template('feed.xml', listelements=entries, folder='opds.feed_publisher', pagination=pagination)


@@ -188,10 +190,10 @@ def feed_publisherindex():
@requires_basic_auth_if_no_ano
 def feed_publisher(book_id):
    off = request.args.get("offset") or 0
-    entries, __, pagination = fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
-                                             db.Books,
-                                             db.Books.publishers.any(db.Publishers.id == book_id),
-                                             [db.Books.timestamp.desc()])
+    entries, __, pagination = calibre_db.fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
+                                                        db.Books,
+                                                        db.Books.publishers.any(db.Publishers.id == book_id),
+                                                        [db.Books.timestamp.desc()])
    return render_xml_template('feed.xml', entries=entries, pagination=pagination)


@@ -199,10 +201,16 @@ def feed_publisher(book_id):
@requires_basic_auth_if_no_ano
 def feed_categoryindex():
    off = request.args.get("offset") or 0
-    entries = db.session.query(db.Tags).join(db.books_tags_link).join(db.Books).filter(common_filters())\
-        .group_by(text('books_tags_link.tag')).order_by(db.Tags.name).offset(off).limit(config.config_books_per_page)
+    entries = calibre_db.session.query(db.Tags)\
+        .join(db.books_tags_link)\
+        .join(db.Books)\
+        .filter(calibre_db.common_filters())\
+        .group_by(text('books_tags_link.tag'))\
+        .order_by(db.Tags.name)\
+        .offset(off)\
+        .limit(config.config_books_per_page)
    pagination = Pagination((int(off) / (int(config.config_books_per_page)) + 1), config.config_books_per_page,
-                            len(db.session.query(db.Tags).all()))
+                            len(calibre_db.session.query(db.Tags).all()))
    return render_xml_template('feed.xml', listelements=entries, folder='opds.feed_category', pagination=pagination)


@@ -210,10 +218,10 @@ def feed_categoryindex():
@requires_basic_auth_if_no_ano
 def feed_category(book_id):
    off = request.args.get("offset") or 0
-    entries, __, pagination = fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
-                                             db.Books,
-                                             db.Books.tags.any(db.Tags.id == book_id),
-                                             [db.Books.timestamp.desc()])
+    entries, __, pagination = calibre_db.fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
+                                                        db.Books,
+                                                        db.Books.tags.any(db.Tags.id == book_id),
+                                                        [db.Books.timestamp.desc()])
    return render_xml_template('feed.xml', entries=entries, pagination=pagination)


@@ -221,10 +229,15 @@ def feed_category(book_id):
@requires_basic_auth_if_no_ano
 def feed_seriesindex():
    off = request.args.get("offset") or 0
-    entries = db.session.query(db.Series).join(db.books_series_link).join(db.Books).filter(common_filters())\
-        .group_by(text('books_series_link.series')).order_by(db.Series.sort).offset(off).all()
+    entries = calibre_db.session.query(db.Series)\
+        .join(db.books_series_link)\
+        .join(db.Books)\
+        .filter(calibre_db.common_filters())\
+        .group_by(text('books_series_link.series'))\
+        .order_by(db.Series.sort)\
+        .offset(off).all()
    pagination = Pagination((int(off) / (int(config.config_books_per_page)) + 1), config.config_books_per_page,
-                            len(db.session.query(db.Series).all()))
+                            len(calibre_db.session.query(db.Series).all()))
    return render_xml_template('feed.xml', listelements=entries, folder='opds.feed_series', pagination=pagination)


@@ -232,10 +245,10 @@ def feed_seriesindex():
@requires_basic_auth_if_no_ano
 def feed_series(book_id):
    off = request.args.get("offset") or 0
-    entries, __, pagination = fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
-                                             db.Books,
-                                             db.Books.series.any(db.Series.id == book_id),
-                                             [db.Books.series_index])
+    entries, __, pagination = calibre_db.fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
+                                                        db.Books,
+                                                        db.Books.series.any(db.Series.id == book_id),
+                                                        [db.Books.series_index])
    return render_xml_template('feed.xml', entries=entries, pagination=pagination)


@@ -243,10 +256,13 @@ def feed_series(book_id):
@requires_basic_auth_if_no_ano
 def feed_ratingindex():
    off = request.args.get("offset") or 0
-    entries = db.session.query(db.Ratings, func.count('books_ratings_link.book').label('count'),
+    entries = calibre_db.session.query(db.Ratings, func.count('books_ratings_link.book').label('count'),
                               (db.Ratings.rating / 2).label('name')) \
-        .join(db.books_ratings_link).join(db.Books).filter(common_filters()) \
-        .group_by(text('books_ratings_link.rating')).order_by(db.Ratings.rating).all()
+        .join(db.books_ratings_link)\
+        .join(db.Books)\
+        .filter(calibre_db.common_filters()) \
+        .group_by(text('books_ratings_link.rating'))\
+        .order_by(db.Ratings.rating).all()

    pagination = Pagination((int(off) / (int(config.config_books_per_page)) + 1), config.config_books_per_page,
                            len(entries))
@@ -260,10 +276,10 @@ def feed_ratingindex():
@requires_basic_auth_if_no_ano
 def feed_ratings(book_id):
    off = request.args.get("offset") or 0
-    entries, __, pagination = fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
-                                             db.Books,
-                                             db.Books.ratings.any(db.Ratings.id == book_id),
-                                             [db.Books.timestamp.desc()])
+    entries, __, pagination = calibre_db.fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
+                                                        db.Books,
+                                                        db.Books.ratings.any(db.Ratings.id == book_id),
+                                                        [db.Books.timestamp.desc()])
    return render_xml_template('feed.xml', entries=entries, pagination=pagination)


@@ -271,8 +287,10 @@ def feed_ratings(book_id):
@requires_basic_auth_if_no_ano
 def feed_formatindex():
    off = request.args.get("offset") or 0
-    entries = db.session.query(db.Data).join(db.Books).filter(common_filters()) \
-        .group_by(db.Data.format).order_by(db.Data.format).all()
+    entries = calibre_db.session.query(db.Data).join(db.Books)\
+        .filter(calibre_db.common_filters()) \
+        .group_by(db.Data.format)\
+        .order_by(db.Data.format).all()
    pagination = Pagination((int(off) / (int(config.config_books_per_page)) + 1), config.config_books_per_page,
                            len(entries))

@@ -286,10 +304,10 @@ def feed_formatindex():
@requires_basic_auth_if_no_ano
 def feed_format(book_id):
    off = request.args.get("offset") or 0
-    entries, __, pagination = fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
-                                             db.Books,
-                                             db.Books.data.any(db.Data.format == book_id.upper()),
-                                             [db.Books.timestamp.desc()])
+    entries, __, pagination = calibre_db.fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
+                                                        db.Books,
+                                                        db.Books.data.any(db.Data.format == book_id.upper()),
+                                                        [db.Books.timestamp.desc()])
    return render_xml_template('feed.xml', entries=entries, pagination=pagination)


@@ -299,13 +317,13 @@ def feed_format(book_id):
 def feed_languagesindex():
    off = request.args.get("offset") or 0
    if current_user.filter_language() == u"all":
-        languages = speaking_language()
+        languages = calibre_db.speaking_language()
    else:
        try:
            cur_l = LC.parse(current_user.filter_language())
        except UnknownLocaleError:
            cur_l = None
-        languages = db.session.query(db.Languages).filter(
+        languages = calibre_db.session.query(db.Languages).filter(
            db.Languages.lang_code == current_user.filter_language()).all()
        if cur_l:
            languages[0].name = cur_l.get_language_name(get_locale())
@@ -320,10 +338,10 @@ def feed_languagesindex():
@requires_basic_auth_if_no_ano
 def feed_languages(book_id):
    off = request.args.get("offset") or 0
-    entries, __, pagination = fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
-                                             db.Books,
-                                             db.Books.languages.any(db.Languages.id == book_id),
-                                             [db.Books.timestamp.desc()])
+    entries, __, pagination = calibre_db.fill_indexpage((int(off) / (int(config.config_books_per_page)) + 1),
+                                                        db.Books,
+                                                        db.Books.languages.any(db.Languages.id == book_id),
+                                                        [db.Books.timestamp.desc()])
    return render_xml_template('feed.xml', entries=entries, pagination=pagination)


@@ -356,7 +374,7 @@ def feed_shelf(book_id):
        books_in_shelf = ub.session.query(ub.BookShelf).filter(ub.BookShelf.shelf == book_id).order_by(
            ub.BookShelf.order.asc()).all()
        for book in books_in_shelf:
-            cur_book = db.session.query(db.Books).filter(db.Books.id == book.book_id).first()
+            cur_book = calibre_db.get_book(book.book_id)
            result.append(cur_book)
    pagination = Pagination((int(off) / (int(config.config_books_per_page)) + 1), config.config_books_per_page,
                            len(result))
@@ -367,14 +385,18 @@ def feed_shelf(book_id):
@requires_basic_auth_if_no_ano
@download_required
 def opds_download_link(book_id, book_format):
-    return get_download_link(book_id, book_format.lower())
+    if "Kobo" in request.headers.get('User-Agent'):
+        client = "kobo"
+    else:
+        client = ""
+    return get_download_link(book_id, book_format.lower(), client)


@opds.route("/ajax/book/<string:uuid>/<library>")
@opds.route("/ajax/book/<string:uuid>", defaults={'library': ""})
@requires_basic_auth_if_no_ano
 def get_metadata_calibre_companion(uuid, library):
-    entry = db.session.query(db.Books).filter(db.Books.uuid.like("%" + uuid + "%")).first()
+    entry = calibre_db.session.query(db.Books).filter(db.Books.uuid.like("%" + uuid + "%")).first()
    if entry is not None:
        js = render_template('json.txt', entry=entry)
        response = make_response(js)
@@ -386,8 +408,7 @@ def get_metadata_calibre_companion(uuid, library):

 def feed_search(term):
    if term:
-        term = term.strip().lower()
-        entries = get_search_results(term)
+        entries = calibre_db.get_search_results(term)
        entriescount = len(entries) if len(entries) > 0 else 1
        pagination = Pagination(1, entriescount, entriescount)
        return render_xml_template('feed.xml', searchterm=term, entries=entries, pagination=pagination)
@@ -77,6 +77,7 @@ class ReverseProxied(object):
        servr = environ.get('HTTP_X_FORWARDED_HOST', '')
        if servr:
            environ['HTTP_HOST'] = servr
+            self.proxied = True
        return self.app(environ, start_response)

    @property
@@ -72,7 +72,11 @@ class WebServer(object):
        if config.config_access_log:
            log_name = "gevent.access" if _GEVENT else "tornado.access"
            formatter = logger.ACCESS_FORMATTER_GEVENT if _GEVENT else logger.ACCESS_FORMATTER_TORNADO
-            self.access_logger = logger.create_access_log(config.config_access_logfile, log_name, formatter)
+            self.access_logger, logfile = logger.create_access_log(config.config_access_logfile, log_name, formatter)
+            if logfile != config.config_access_logfile:
+                log.warning("Accesslog path %s not valid, falling back to default", config.config_access_logfile)
+                config.config_access_logfile = logfile
+                config.save()
        else:
            if not _GEVENT:
                logger.get('tornado.access').disabled = True
@@ -190,12 +194,15 @@ class WebServer(object):
        os.execv(sys.executable, arguments)
        return True

-    def _killServer(self, ignored_signum, ignored_frame):
+    def _killServer(self, __, ___):
        self.stop()

    def stop(self, restart=False):
        from . import updater_thread
        updater_thread.stop()
+        from . import calibre_db
+        calibre_db.stop()
+

        log.info("webserver stop (restart=%s)", restart)
        self.restart = restart
@@ -27,7 +27,10 @@ except ImportError:
    from urllib.parse import unquote

 from flask import json
-from .. import logger as log
+from .. import logger
+
+
+log = logger.create()


 def b64encode_json(json_data):
@@ -45,7 +48,8 @@ def to_epoch_timestamp(datetime_object):
 def get_datetime_from_json(json_object, field_name):
    try:
        return datetime.utcfromtimestamp(json_object[field_name])
-    except KeyError:
+    except (KeyError, OSError, OverflowError):
+        # OSError is thrown on Windows if timestamp is <1970 or >2038
        return datetime.min


@@ -20,7 +20,10 @@ from __future__ import division, print_function, unicode_literals
 import time
 from functools import reduce

-from goodreads.client import GoodreadsClient
+try:
+    from goodreads.client import GoodreadsClient
+except ImportError:
+    from betterreads.client import GoodreadsClient

 try: import Levenshtein
 except ImportError: Levenshtein = False
@@ -95,8 +98,12 @@ def get_other_books(author_info, library_books=None):
    for book in author_info.books:
        if book.isbn in identifiers:
            continue
-        if book.gid["#text"] in identifiers:
-            continue
+        if isinstance(book.gid, int):
+            if book.gid in identifiers:
+                continue
+        else:
+            if book.gid["#text"] in identifiers:
+                continue

        if Levenshtein and library_titles:
            goodreads_title = book._book_dict['title_without_series']
@@ -64,9 +64,11 @@ def init_app(app, config):
    app.config['LDAP_OPENLDAP'] = bool(config.config_ldap_openldap)
    app.config['LDAP_GROUP_OBJECT_FILTER'] = config.config_ldap_group_object_filter
    app.config['LDAP_GROUP_MEMBERS_FIELD'] = config.config_ldap_group_members_field
-    # app.config['LDAP_CUSTOM_OPTIONS'] = {'OPT_NETWORK_TIMEOUT': 10}

-    _ldap.init_app(app)
+    try:
+        _ldap.init_app(app)
+    except RuntimeError as e:
+        log.error(e)


 def get_object_details(user=None, group=None, query_filter=None, dn_only=False):
@@ -27,10 +27,10 @@ from flask import Blueprint, request, flash, redirect, url_for
 from flask_babel import gettext as _
 from flask_login import login_required, current_user
 from sqlalchemy.sql.expression import func
+from sqlalchemy.exc import OperationalError, InvalidRequestError

-from . import logger, ub, searched_ids, db
-from .web import render_title_template
-from .helper import common_filters
+from . import logger, ub, searched_ids, calibre_db
+from .web import login_required_if_no_ano, render_title_template


 shelf = Blueprint('shelf', __name__)
@@ -92,8 +92,16 @@ def add_to_shelf(shelf_id, book_id):

    shelf.books.append(ub.BookShelf(shelf=shelf.id, book_id=book_id, order=maxOrder + 1))
    shelf.last_modified = datetime.utcnow()
-    ub.session.merge(shelf)
-    ub.session.commit()
+    try:
+        ub.session.merge(shelf)
+        ub.session.commit()
+    except (OperationalError, InvalidRequestError):
+        ub.session.rollback()
+        flash(_(u"Settings DB is not Writeable"), category="error")
+        if "HTTP_REFERER" in request.environ:
+            return redirect(request.environ["HTTP_REFERER"])
+        else:
+            return redirect(url_for('web.index'))
    if not xhr:
        flash(_(u"Book has been added to shelf: %(sname)s", sname=shelf.name), category="success")
        if "HTTP_REFERER" in request.environ:
@@ -144,9 +152,13 @@ def search_to_shelf(shelf_id):
            maxOrder = maxOrder + 1
            shelf.books.append(ub.BookShelf(shelf=shelf.id, book_id=book, order=maxOrder))
        shelf.last_modified = datetime.utcnow()
-        ub.session.merge(shelf)
-        ub.session.commit()
-        flash(_(u"Books have been added to shelf: %(sname)s", sname=shelf.name), category="success")
+        try:
+            ub.session.merge(shelf)
+            ub.session.commit()
+            flash(_(u"Books have been added to shelf: %(sname)s", sname=shelf.name), category="success")
+        except (OperationalError, InvalidRequestError):
+            ub.session.rollback()
+            flash(_(u"Settings DB is not Writeable"), category="error")
    else:
        flash(_(u"Could not add books to shelf: %(sname)s", sname=shelf.name), category="error")
    return redirect(url_for('web.index'))
@@ -181,10 +193,17 @@ def remove_from_shelf(shelf_id, book_id):
                return redirect(url_for('web.index'))
            return "Book already removed from shelf", 410

-        ub.session.delete(book_shelf)
-        shelf.last_modified = datetime.utcnow()
-        ub.session.commit()
-
+        try:
+            ub.session.delete(book_shelf)
+            shelf.last_modified = datetime.utcnow()
+            ub.session.commit()
+        except (OperationalError, InvalidRequestError):
+            ub.session.rollback()
+            flash(_(u"Settings DB is not Writeable"), category="error")
+            if "HTTP_REFERER" in request.environ:
+                return redirect(request.environ["HTTP_REFERER"])
+            else:
+                return redirect(url_for('web.index'))
        if not xhr:
            flash(_(u"Book has been removed from shelf: %(sname)s", sname=shelf.name), category="success")
            if "HTTP_REFERER" in request.environ:
@@ -236,7 +255,11 @@ def create_shelf():
                ub.session.commit()
                flash(_(u"Shelf %(title)s created", title=to_save["title"]), category="success")
                return redirect(url_for('shelf.show_shelf', shelf_id=shelf.id))
+            except (OperationalError, InvalidRequestError):
+                ub.session.rollback()
+                flash(_(u"Settings DB is not Writeable"), category="error")
            except Exception:
+                ub.session.rollback()
                flash(_(u"There was an error"), category="error")
        return render_title_template('shelf_edit.html', shelf=shelf, title=_(u"Create a Shelf"), page="shelfcreate")
    else:
@@ -281,7 +304,11 @@ def edit_shelf(shelf_id):
            try:
                ub.session.commit()
                flash(_(u"Shelf %(title)s changed", title=to_save["title"]), category="success")
+            except (OperationalError, InvalidRequestError):
+                ub.session.rollback()
+                flash(_(u"Settings DB is not Writeable"), category="error")
            except Exception:
+                ub.session.rollback()
                flash(_(u"There was an error"), category="error")
        return render_title_template('shelf_edit.html', shelf=shelf, title=_(u"Edit a shelf"), page="shelfedit")
    else:
@@ -299,16 +326,22 @@ def delete_shelf_helper(cur_shelf):
    log.info("successfully deleted %s", cur_shelf)


+
@shelf.route("/shelf/delete/<int:shelf_id>")
@login_required
 def delete_shelf(shelf_id):
    cur_shelf = ub.session.query(ub.Shelf).filter(ub.Shelf.id == shelf_id).first()
-    delete_shelf_helper(cur_shelf)
+    try:
+        delete_shelf_helper(cur_shelf)
+    except (OperationalError, InvalidRequestError):
+        ub.session.rollback()
+        flash(_(u"Settings DB is not Writeable"), category="error")
    return redirect(url_for('web.index'))


@shelf.route("/shelf/<int:shelf_id>", defaults={'shelf_type': 1})
@shelf.route("/shelf/<int:shelf_id>/<int:shelf_type>")
+@login_required_if_no_ano
 def show_shelf(shelf_type, shelf_id):
    shelf = ub.session.query(ub.Shelf).filter(ub.Shelf.id == shelf_id).first()

@@ -320,15 +353,19 @@ def show_shelf(shelf_type, shelf_id):
        books_in_shelf = ub.session.query(ub.BookShelf).filter(ub.BookShelf.shelf == shelf_id)\
            .order_by(ub.BookShelf.order.asc()).all()
        for book in books_in_shelf:
-            cur_book = db.session.query(db.Books).filter(db.Books.id == book.book_id).filter(common_filters()).first()
+            cur_book = calibre_db.get_filtered_book(book.book_id)
            if cur_book:
                result.append(cur_book)
            else:
-                cur_book = db.session.query(db.Books).filter(db.Books.id == book.book_id).first()
+                cur_book = calibre_db.get_book(book.book_id)
                if not cur_book:
                    log.info('Not existing book %s in %s deleted', book.book_id, shelf)
-                    ub.session.query(ub.BookShelf).filter(ub.BookShelf.book_id == book.book_id).delete()
-                    ub.session.commit()
+                    try:
+                        ub.session.query(ub.BookShelf).filter(ub.BookShelf.book_id == book.book_id).delete()
+                        ub.session.commit()
+                    except (OperationalError, InvalidRequestError):
+                        ub.session.rollback()
+                        flash(_(u"Settings DB is not Writeable"), category="error")
        return render_title_template(page, entries=result, title=_(u"Shelf: '%(name)s'", name=shelf.name),
                                     shelf=shelf, page="shelf")
    else:
@@ -348,7 +385,11 @@ def order_shelf(shelf_id):
            setattr(book, 'order', to_save[str(book.book_id)])
            counter += 1
            # if order diffrent from before -> shelf.last_modified = datetime.utcnow()
-        ub.session.commit()
+        try:
+            ub.session.commit()
+        except (OperationalError, InvalidRequestError):
+            ub.session.rollback()
+            flash(_(u"Settings DB is not Writeable"), category="error")

    shelf = ub.session.query(ub.Shelf).filter(ub.Shelf.id == shelf_id).first()
    result = list()
@@ -356,7 +397,7 @@ def order_shelf(shelf_id):
        books_in_shelf2 = ub.session.query(ub.BookShelf).filter(ub.BookShelf.shelf == shelf_id) \
            .order_by(ub.BookShelf.order.asc()).all()
        for book in books_in_shelf2:
-            cur_book = db.session.query(db.Books).filter(db.Books.id == book.book_id).filter(common_filters()).first()
+            cur_book = calibre_db.get_filtered_book(book.book_id)
            if cur_book:
                result.append({'title': cur_book.title,
                               'id': cur_book.id,
@@ -364,7 +405,7 @@ def order_shelf(shelf_id):
                               'series': cur_book.series,
                               'series_index': cur_book.series_index})
            else:
-                cur_book = db.session.query(db.Books).filter(db.Books.id == book.book_id).first()
+                cur_book = calibre_db.get_book(book.book_id)
                result.append({'title': _('Hidden Book'),
                               'id': cur_book.id,
                               'author': [],
@@ -0,0 +1,17 @@
+body.serieslist.grid-view div.container-fluid>div>div.col-sm-10:before{
+    display: none;
+}
+
+.cover .badge{
+    position: absolute;
+    top: 0;
+    right: 0;
+    background-color: #cc7b19;
+    border-radius: 0;
+    padding: 0 8px;
+    box-shadow: 0 0 4px rgba(0,0,0,.6);
+    line-height: 24px;
+}
+.cover{
+    box-shadow: 0 0 4px rgba(0,0,0,.6);
+}
@@ -109,7 +109,7 @@ a { color: #45b29d; }

 .container-fluid .book .cover img {
  border: 1px solid #fff;
-  box-sizeing: border-box;
+  box-sizing: border-box;
  height: 100%;
  bottom: 0;
  position: absolute;
@@ -133,8 +133,9 @@ a { color: #45b29d; }
 }

 .container-fluid .book .meta .rating { margin-top: 5px; }
-.rating .glyphicon-star { color: #999; }
-.rating .glyphicon-star.good { color: #45b29d; }
+.rating .glyphicon-star-empty { color: #444; }
+.rating .glyphicon-star.good { color: #444; }
+.rating-clear .glyphicon-remove { color: #333  }

 .container-fluid .author .author-hidden, .container-fluid .author .author-hidden-divider { display: none; }

@@ -165,7 +166,7 @@ span.glyphicon.glyphicon-tags {

 .container-fluid .single .cover img {
  border: 1px solid #fff;
-  box-sizeing: border-box;
+  box-sizing: border-box;
  -webkit-box-shadow: 0 5px 8px -6px #777;
  -moz-box-shadow: 0 5px 8px -6px #777;
  box-shadow: 0 5px 8px -6px #777;
@@ -174,6 +175,12 @@ span.glyphicon.glyphicon-tags {
 .navbar-default .navbar-toggle .icon-bar {background-color: #000; }
 .navbar-default .navbar-toggle {border-color: #000; }
 .cover { margin-bottom: 10px; }
+.cover .badge{
+   position: absolute;
+   top: 2px;
+   left: 2px;
+   background-color: #777;
+}
 .cover-height { max-height: 100px;}

 .col-sm-2 a .cover-small {
@@ -207,7 +214,7 @@ span.glyphicon.glyphicon-tags {
 .panel-body {background-color: #f5f5f5; }
 .spinner {margin: 0 41%; }
 .spinner2 {margin: 0 41%; }
-
+.intend-form { margin-left:20px; }
 table .bg-dark-danger {background-color: #d9534f; color: #fff; }
 table .bg-dark-danger a {color: #fff; }
 table .bg-dark-danger:hover {background-color: #c9302c; }
@@ -296,3 +303,4 @@ div.log {
  white-space: nowrap;
  padding: 0.5em;
 }
+
@@ -0,0 +1,54 @@
+/* This file is part of the Calibre-Web (https://github.com/janeczku/calibre-web)
+ *    Copyright (C) 2018 OzzieIsaacs
+ *
+ *  This program is free software: you can redistribute it and/or modify
+ *  it under the terms of the GNU General Public License as published by
+ *  the Free Software Foundation, either version 3 of the License, or
+ *  (at your option) any later version.
+ *
+ *  This program is distributed in the hope that it will be useful,
+ *  but WITHOUT ANY WARRANTY; without even the implied warranty of
+ *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ *  GNU General Public License for more details.
+ *
+ *  You should have received a copy of the GNU General Public License
+ *  along with this program. If not, see <http://www.gnu.org/licenses/>.
+ */
+
+var $list = $("#list").isotope({
+    itemSelector: ".book",
+    layoutMode: "fitRows",
+    getSortData: {
+        title: ".title",
+    }
+});
+
+$("#desc").click(function() {
+    $list.isotope({
+        sortBy: "name",
+        sortAscending: true
+    });
+    return;
+});
+
+$("#asc").click(function() {
+    $list.isotope({
+        sortBy: "name",
+        sortAscending: false
+    });
+    return;
+});
+
+$("#all").click(function() {
+    // go through all elements and make them visible
+    $list.isotope({ filter: function() {
+        return true;
+      } })
+});
+
+$(".char").click(function() {
+    var character = this.innerText;
+    $list.isotope({ filter: function() {
+        return this.attributes["data-id"].value.charAt(0).toUpperCase() == character;
+      } })
+});
@@ -74,8 +74,8 @@ $(function () {
            $("#meta-info").html("<ul id=\"book-list\" class=\"media-list\"></ul>");
        }
        if ((ggDone === 3 || (ggDone === 1 && ggResults.length === 0)) &&
-            (dbDone === 3 || (ggDone === 1 && dbResults.length === 0)) &&
-            (cvDone === 3 || (ggDone === 1 && cvResults.length === 0))) {
+            (dbDone === 3 || (dbDone === 1 && dbResults.length === 0)) &&
+            (cvDone === 3 || (cvDone === 1 && cvResults.length === 0))) {
            $("#meta-info").html("<p class=\"text-danger\">" + msg.no_result + "</p>");
            return;
        }
@@ -323,4 +323,17 @@ $(function() {
        $(".discover .row").isotope("layout");
    });

+    $(".update-view").click(function(e) {
+        var target = $(this).data("target");
+        var view = $(this).data("view");
+
+        e.preventDefault();
+        e.stopPropagation();
+        var data = {};
+        data[target] = view;
+        console.debug("Updating view data: ", data);
+        $.post( "/ajax/view", data).done(function( ) {
+            location.reload();
+        });
+    });
 });
@@ -124,7 +124,7 @@ $(function() {
                if (field === 3) {
                    $.ajax ({
                        type: "Post",
-                        data: "id=" + row.id + "&type=" + row.type + "&Element=" + row.Element,
+                        data: "id=" + row.id + "&type=" + row.type + "&Element=" + encodeURIComponent(row.Element),
                        url: path + "/../../ajax/deleterestriction/" + type,
                        async: true,
                        timeout: 900,
@@ -22,7 +22,7 @@ import os
 import subprocess


-def process_open(command, quotes=(), env=None, sout=subprocess.PIPE, serr=subprocess.PIPE):
+def process_open(command, quotes=(), env=None, sout=subprocess.PIPE, serr=subprocess.PIPE, newlines=True):
    # Linux py2.7 encode as list without quotes no empty element for parameters
    # linux py3.x no encode and as list without quotes no empty element for parameters
    # windows py2.7 encode as string with quotes empty element for parameters is okay
@@ -41,12 +41,13 @@ def process_open(command, quotes=(), env=None, sout=subprocess.PIPE, serr=subpro
        else:
            exc_command = [x for x in command]

-    return subprocess.Popen(exc_command, shell=False, stdout=sout, stderr=serr, universal_newlines=True, env=env)
+    return subprocess.Popen(exc_command, shell=False, stdout=sout, stderr=serr, universal_newlines=newlines, env=env)


 def process_wait(command, serr=subprocess.PIPE):
    # Run command, wait for process to terminate, and return an iterator over lines of its output.
-    p = process_open(command, serr=serr)
+    newlines = os.name != 'nt'
+    p = process_open(command, serr=serr, newlines=newlines)
    p.wait()
    for line in p.stdout.readlines():
        if isinstance(line, bytes):
@@ -13,11 +13,14 @@
            <th>{{_('E-mail Address')}}</th>
            <th>{{_('Send to Kindle E-mail Address')}}</th>
            <th>{{_('Downloads')}}</th>
-            <th class="hidden-xs">{{_('Admin')}}</th>
-            <th class="hidden-xs">{{_('Download')}}</th>
-            <th class="hidden-xs">{{_('View Books')}}</th>
-            <th class="hidden-xs">{{_('Upload')}}</th>
-            <th class="hidden-xs">{{_('Edit')}}</th>
+            <th class="hidden-xs ">{{_('Admin')}}</th>
+            <th class="hidden-xs hidden-sm">{{_('Password')}}</th>
+            <th class="hidden-xs hidden-sm">{{_('Upload')}}</th>
+            <th class="hidden-xs hidden-sm">{{_('Download')}}</th>
+            <th class="hidden-xs hidden-sm hidden-md">{{_('View Books')}}</th>
+            <th class="hidden-xs hidden-sm hidden-md">{{_('Edit')}}</th>
+            <th class="hidden-xs hidden-sm hidden-md">{{_('Delete')}}</th>
+            <th class="hidden-xs hidden-sm hidden-md">{{_('Public Shelf')}}</th>
        </tr>
        {% for user in allUser %}
          {% if not user.role_anonymous() or config.config_anonbrowse %}
@@ -27,10 +30,13 @@
            <td>{{user.kindle_mail}}</td>
            <td>{{user.downloads.count()}}</td>
            <td class="hidden-xs">{{ display_bool_setting(user.role_admin()) }}</td>
-            <td class="hidden-xs">{{ display_bool_setting(user.role_download()) }}</td>
-            <td class="hidden-xs">{{ display_bool_setting(user.role_viewer()) }}</td>
-            <td class="hidden-xs">{{ display_bool_setting(user.role_upload()) }}</td>
-            <td class="hidden-xs">{{ display_bool_setting(user.role_edit()) }}</td>
+            <td class="hidden-xs hidden-sm">{{ display_bool_setting(user.role_passwd()) }}</td>
+            <td class="hidden-xs hidden-sm">{{ display_bool_setting(user.role_upload()) }}</td>
+            <td class="hidden-xs hidden-sm">{{ display_bool_setting(user.role_download()) }}</td>
+            <td class="hidden-xs hidden-sm hidden-md">{{ display_bool_setting(user.role_viewer()) }}</td>
+            <td class="hidden-xs hidden-sm hidden-md">{{ display_bool_setting(user.role_edit()) }}</td>
+            <td class="hidden-xs hidden-sm hidden-md">{{ display_bool_setting(user.role_delete_books()) }}</td>
+            <td class="hidden-xs hidden-sm hidden-md">{{ display_bool_setting(user.role_edit_shelfs()) }}</td>
          </tr>
          {% endif %}
        {% endfor %}
@@ -82,6 +88,12 @@
            <div class="col-xs-6 col-sm-6">{{_('Port')}}</div>
            <div class="col-xs-6 col-sm-6">{{config.config_port}}</div>
        </div>
+        {% if feature_support['kobo'] and config.config_port != config.config_external_port %}
+        <div class="row">
+          <div class="col-xs-6 col-sm-6">{{_('External Port')}}</div>
+          <div class="col-xs-6 col-sm-6">{{config.config_external_port}}</div>
+        </div>
+        {% endif %}
      </div>
      <div class="col-xs-12 col-sm-6">
        <div class="row">
--- a/Show More
+++ b/Show More