5.4. Server#
The server has a central function in the vantage6 architecture. It stores in the database which organizations, collaborations, users, etc. exist. It allows the users and nodes to authenticate and subsequently interact through the API the server hosts. Finally, it also communicates with authenticated nodes and users via the socketIO server that is run here.
5.4.1. Main server class#
- class ServerApp(ctx)#
Vantage6 server instance.
- Variables:
ctx (ServerContext) – Context object that contains the configuration of the server.
- configure_api()#
Define global API output and its structure.
- configure_flask()#
Configure the Flask settings of the vantage6 server.
- configure_jwt()#
Configure JWT authentication.
- static configure_logging()#
Set third party loggers to a warning level
- load_resources()#
Import the modules containing API resources.
- start()#
Start the server.
Before server is really started, some database settings are checked and (re)set where appropriate.
5.4.2. Starting the server#
- run_server(config, environment='prod', system_folders=True)#
Run a vantage6 server.
- Parameters:
config (str) – Configuration file path
environment (str) – Configuration environment to use.
system_folders (bool) – Whether to use system or user folders. Default is True.
- Returns:
A running instance of the vantage6 server
- Return type:
Warning
Note that the run_server
function is normally not used directly to
start the server, but is used as utility function in places that start the
server. The recommended way to start a server is using uWSGI as is done in
vserver start
.
5.4.3. Permission management#
- class Scope(value)#
Enumerator of all available scopes
- COLLABORATION = 'col'#
- GLOBAL = 'glo'#
- ORGANIZATION = 'org'#
- OWN = 'own'#
- class Operation(value)#
Enumerator of all available operations
- CREATE = 'c'#
- DELETE = 'd'#
- EDIT = 'e'#
- VIEW = 'v'#
- class RuleCollection(name)#
Class that tracks a set of all rules for a certain resource name
- Parameters:
name (str) – Name of the resource endpoint (e.g. node, organization, user)
- class PermissionManager#
Loads the permissions and syncs rules in database with rules defined in the code
- appender(name)#
Add a module’s rules to the rule collection
- Parameters:
name (str) – The name of the module whose rules are to be registered
- Returns:
A callable
register_rule
function- Return type:
Callable
- assign_rule_to_container(resource, scope, operation)#
Assign a rule to the container role.
- static assign_rule_to_fixed_role(fixedrole, resource, scope, operation)#
Attach a rule to a fixed role (not adjustable by users).
- assign_rule_to_node(resource, scope, operation)#
Assign a rule to the Node role.
- assign_rule_to_root(name, scope, operation)#
Assign a rule to the root role.
- Return type:
None
- resource: str
Resource that the rule applies to
- scope: Scope
Scope that the rule applies to
- operation: Operation
Operation that the rule applies to
- collection(name)#
Get a RuleCollection object. If it doesn’t exist yet, it will be created.
- Parameters:
name (str) – Name of the module whose RuleCollection is to be obtained or created
- Returns:
The collection of rules belonging to the module name
- Return type:
- load_rules_from_resources()#
Collect all permission rules from all registered API resources
- Return type:
None
- register_rule(resource, scope, operation, description=None, assign_to_node=False, assign_to_container=False)#
Register a permission rule in the database.
If a rule already exists, nothing is done. This rule can be used in API endpoints to determine if a user, node or container can do a certain operation in a certain scope.
- Parameters:
resource (str) – API resource that the rule applies to
scope (Scope) – Scope of the rule
operation (Operation) – Operation of the rule
description (String, optional) – Human readable description where the rule is used for, by default None
assign_to_node (bool, optional) – Whether rule should be assigned to the node role or not. Default False
assign_to_container (bool, optional) – Whether rule should be assigned to the container role or not. Default False
- Return type:
None
- static rule_exists_in_db(name, scope, operation)#
Check if the rule exists in the DB.
5.4.4. Socket functionality#
- class DefaultSocketNamespace(namespace=None)#
This is the default SocketIO namespace. It is used for all the long-running socket communication between the server and the clients. The clients of the socket connection are nodes and users.
When socket communication is received from one of the clients, the functions in this class are called to execute the corresponding action.
- on_algorithm_status_change(data)#
An algorithm container has changed its status. This status change may be that the algorithm has finished, crashed, etc. Here we notify the collaboration of the change.
- Parameters:
data (Dict) –
Dictionary containing parameters on the updated algorithm status. It should look as follows:
- {
# node_id where algorithm container was running “node_id”: 1, # new status of algorithm container “status”: “active”, # result_id for which the algorithm was running “result_id”: 1, # collaboration_id for which the algorithm was running “collaboration_id”: 1
}
- Return type:
None
- on_connect()#
A new incoming connection request from a client.
New connections are authenticated using their JWT authorization token which is obtained from the REST API. A session is created for each connected client, and lives as long as the connection is active. Each client is assigned to rooms based on their permissions.
Nodes that are connecting are also set to status ‘online’.
- Return type:
None
Note
Note that reconnecting clients are treated the same as new clients.
- on_disconnect()#
Client that disconnects is removed from all rooms they were in.
If nodes disconnect, their status is also set to offline and users may be alerted to that. Also, any information on the node (e.g. configuration) is removed from the database.
- Return type:
None
- on_error(e)#
An receiving an error from a client, log it.
- Parameters:
e (str) – Error message that is being displayed in the server log
- Return type:
None
- on_message(message)#
On receiving a message from a client, log it.
- Parameters:
message (str) – Message that is going to be displayed in the server log
- Return type:
None
- on_node_info_update(node_config)#
A node sends information about its configuration and other properties. Store this in the database for the duration of the node’s session.
- Parameters:
node_config (dict) – Dictionary containing the node’s configuration.
- Return type:
None
5.4.5. API endpoints#
Warning
The API endpoints are also documented on the /apidocs
endpoint of the
server (e.g. https://petronas.vantage6.ai/apidocs
). We are therefore
not including the API documentation here. Instead, we merely list the
supporting functions and classes.
5.4.6. SQLAlchemy models#
Helper (base) classes#
- class Database(*args, **kwargs)#
Database class that is used to connect to the database and create the database session.
The database is created as a singleton, so that it can be destroyed (as opposed to a module). This is especially useful when creating unit tests in which we want fresh databases every now and then.
- add_col_to_table(column, table_cls)#
Database operation to add column to Table
- Parameters:
column (Column) – The SQLAlchemy model column that is to be added
table_cls (Table) – The SQLAlchemy table to which the column is to be added
- Return type:
None
- add_missing_columns()#
Check database tables to see if columns are missing that are described in the SQLAlchemy models, and add the missing columns
- Return type:
None
- clear_data()#
Clear all data from the database.
- close()#
Delete all tables and close the database connection. Only used for unit testing.
- connect(uri='sqlite:////tmp/test.db', allow_drop_all=False)#
Connect to the database.
- Parameters:
uri (str) – URI of the database. Defaults to a sqlite database in /tmp.
allow_drop_all (bool, optional) – If True, the database can be dropped. Defaults to False.
- drop_all()#
Drop all tables in the database.
- get_non_existing_columns(table_cls, table_name)#
Return a list of columns that are defined in the SQLAlchemy model, but are not present in the database
- Parameters:
table_cls (Table) – The table that is evaluated
table_name (str) – The name of the table
- Returns:
List of SQLAlchemy Column objects that are present in the model, but not in the database
- Return type:
List[Column]
- static is_column_missing(column, column_names, table_name)#
Check if column is missing in the table
- Parameters:
column (Column) – The column that is evaluated
column_names (List[str]) – A list of all column names in the table
table_name (str) – The name of the table the column resides in
- Returns:
True if column is not in the table or a parent table
- Return type:
boolean
- class DatabaseSessionManager#
Class to manage DB sessions.
There are 2 different ways a session can be obtained. Either a session used within a request or a session used elsewhere (e.g. socketIO event, iPython or within the application itself).
In case of the Flask request, the session is stored in the flask global g. Then, it can be accessed in every endpoint.
In all other cases the session is attached to the db module.
- static clear_session()#
Clear the session. If we are in a flask request, the session is cleared from the flask global g. Otherwise, the session is removed from the db module.
- Return type:
None
- static get_session()#
Get a session. Creates a new session if none exists.
- Returns:
A database session
- Return type:
Session
- static in_flask_request()#
Check if we are in a flask request.
- Returns:
True if we are in a flask request, False otherwise
- Return type:
boolean
- static new_session()#
Create a new session. If we are in a flask request, the session is stored in the flask global g. Otherwise, the session is stored in the db module.
- Return type:
None
- class ModelBase#
Declarative base that defines default attributes. All data models inherit from this class.
- delete()#
Delete the object from the database.
- Return type:
None
- classmethod get(id_=None)#
Get a single object by its id, or a list of objects when no id is specified.
- Parameters:
id (int, optional) – The id of the object to get. If not specified, return all.
- classmethod help()#
Print a help message for the class.
- Return type:
None
- save()#
Save the object to the database.
- Return type:
None
Database models for the API resources#
- class AlgorithmPort(**kwargs)#
Table that describes which algorithms are reachable via which ports
Each algorithm with a VPN connection can claim multiple ports via the Dockerfile
EXPOSE
andLABEL
commands. These claims are saved in this table. Each algorithm container belongs to a singleResult
.
- class Authenticatable(**kwargs)#
Parent table of database entities that can authenticate.
Entities that can authenticate are nodes and users. Containers can also authenticate but these are authenticated indirectly through the nodes.
- static hash(secret)#
Hash a secret using bcrypt.
- Parameters:
secret (str) – Secret to be hashed
- Returns:
Hashed secret
- Return type:
str
- class Collaboration(**kwargs)#
Table that describes which collaborations are available.
Collaborations are combinations of one or more organizations that do studies together. Each
Organization
has aNode
for each collaboration that it is part of. Within a collaboration multipleTask
can be executed.- Variables:
name (str) – Name of the collaboration
encrypted (bool) – Whether the collaboration is encrypted or not
organizations (list[
Organization
]) – List of organizations that are part of this collaborationnodes (list[
Node
]) – List of nodes that are part of this collaborationtasks (list[
Task
]) – List of tasks that are part of this collaboration
- classmethod find_by_name(name)#
Find
Collaboration
by its name.Note
If multiple collaborations share the same name, the first collaboration found is returned.
- Parameters:
name (str) – Name of the collaboration
- Returns:
Collaboration with the given name, or None if no collaboration with the given name exists.
- Return type:
Union[Collaboration, None]
- get_node_from_organization(organization)#
Returns the node that is part of the given
Organization
.- Parameters:
organization (Organization) – Organization
- Returns:
Node for the given organization for this collaboration, or None if there is no node for the given organization.
- Return type:
Union[
Node
, None]
- get_nodes_from_organizations(ids)#
Returns a subset of nodes that are part of the given organizations.
- Parameters:
ids (list[int]) – List of organization ids
- Returns:
List of nodes that are part of the given organizations
- Return type:
list[
Node
]
- get_organization_ids()#
Returns a list of organization ids that are part of this collaboration.
- Returns:
List of organization ids
- Return type:
list[int]
- get_task_ids()#
Returns a list of task ids that are part of this collaboration.
- Returns:
List of task ids
- Return type:
list[int]
- classmethod name_exists(name)#
Check if a collaboration with the given name exists.
- Parameters:
name (str) – Name of the collaboration
- Returns:
True if a collaboration with the given name exists, else False
- Return type:
bool
- class Node(**kwargs)#
Bases:
Authenticatable
Table that contains all registered nodes.
- Variables:
id (int) – Primary key
name (str) – Name of the node
api_key (str) – API key of the node
collaboration (
Collaboration
) – Collaboration that the node belongs toorganization (
Organization
) – Organization that the node belongs to
- check_key(key)#
Checks if the provided key matches the stored key.
- Parameters:
key (str) – The key to check
- Returns:
True if the provided key matches the stored key, False otherwise
- Return type:
bool
- classmethod exists(organization_id, collaboration_id)#
Check if a node exists for the given organization and collaboration.
- Parameters:
organization_id (int) – The id of the organization
collaboration_id (int) – The id of the collaboration
- Returns:
True if a node exists for the given organization and collaboration, False otherwise.
- Return type:
bool
- class Organization(**kwargs)#
Table that describes which organizations are available.
An organization is the legal entity that plays a central role in managing distributed tasks. Each organization contains a public key which other organizations can use to send encrypted messages that only this organization can read.
- Variables:
name (str) – Name of the organization
domain (str) – Domain of the organization
address1 (str) – Address of the organization
address2 (str) – Address of the organization
zipcode (str) – Zipcode of the organization
country (str) – Country of the organization
_public_key (bytes) – Public key of the organization
collaborations (list[Collaboration]) – List of collaborations that this organization is part of
results (list[
Result
]) – List of results that are part of this organizationnodes (list[
Node
]) – List of nodes that are part of this organizationusers (list[User]) – List of users that are part of this organization
created_tasks (list[Task]) – List of tasks that are created by this organization
roles (list[Role]) –
- classmethod get_by_name(name)#
Returns the organization with the given name.
- Parameters:
name (str) – Name of the organization
- Returns:
Organization with the given name if it exists, otherwise None
- Return type:
Organization | None
- get_result_ids()#
Returns a list of result ids that are part of this organization.
- Returns:
List of result ids
- Return type:
list[int]
- public_key#
Returns the public key of the organization.
- Returns:
Public key of the organization. Empty string if no public key is set.
- Return type:
str
- class Result(**kwargs)#
Table that describes which results are available. A Result is the description of a Task as executed by a Node.
The result (and the input) is encrypted and can be only read by the intended receiver of the message.
- Variables:
input (str) – Input data of the task
task_id (int) – Id of the task that was executed
organization_id (int) – Id of the organization that executed the task
result (str) – Result of the task
assigned_at (datetime) – Time when the task was assigned to the node
started_at (datetime) – Time when the task was started
finished_at (datetime) – Time when the task was finished
status (str) – Status of the task
log (str) – Log of the task
task (Task) – Task that was executed
organization (Organization) – Organization that executed the task
ports (list[AlgorithmPort]) – List of ports that are part of this result
- complete#
Returns whether the algorithm run has completed or not.
- Returns:
True if the algorithm run has completed, False otherwise.
- Return type:
bool
- class Role(**kwargs)#
Collection of Rules
- Variables:
name (str) – Name of the role
description (str) – Description of the role
organization_id (int) – Id of the organization this role belongs to
rules (List[Rule]) – List of rules that belong to this role
organization (Organization) – Organization this role belongs to
users (List[User]) – List of users that belong to this role
- class Rule(**kwargs)#
Rules to determine permissions in an API endpoint.
A rule gives access to a single type of action with a given operation, scope and resource on which it acts. Note that rules are defined on startup of the server, based on permissions defined in the endpoints. You cannot edit the rules in the database.
- Variables:
- classmethod get_by_(name, scope, operation)#
Get a rule by its name, scope and operation.
- Parameters:
name (str) – Name of the resource on which the rule acts, e.g. ‘node’
scope (str) – Scope of the rule, e.g. ‘organization’
operation (str) – Operation of the rule, e.g. ‘view’
- Returns:
Rule with the given name, scope and operation or None if no rule with the given name, scope and operation exists
- Return type:
Rule | None
- class Task(**kwargs)#
Table that describes all tasks.
A task can contain multiple Results for multiple organizations. The input of the task is different for each organization (due to the encryption). Therefore the input for the task is encrypted for each organization separately. The task originates from an organization to which the results need to be encrypted, therefore the originating organization is also logged
- Variables:
name (str) – Name of the task
description (str) – Description of the task
image (str) – Name of the docker image that needs to be executed
collaboration_id (int) – Id of the collaboration that this task belongs to
run_id (int) – Run id of the task
parent_id (int) – Id of the parent task (if any)
database (str) – Name of the database that needs to be used for this task
initiator_id (int) – Id of the organization that created this task
init_user_id (int) – Id of the user that created this task
collaboration (
Collaboration
) – Collaboration that this task belongs toparent (
Task
) – Parent task (if any)results (list[
Result
]) – List of results that are part of this taskinitiator (
Organization
) – Organization that created this taskinit_user (
User
) – User that created this task
- classmethod next_run_id()#
Get the next available run id for a new task.
- Returns:
Next available run id
- Return type:
int
- results_for_node(node)#
Get all results for a given node.
- Parameters:
node (model.node.Node) – Node for which to get the results
- Returns:
List of results for the given node
- Return type:
List[model.result.Result]
- class User(**kwargs)#
Bases:
Authenticatable
Table to keep track of Users (persons) that can access the system.
Users always belong to an organization and can have certain rights within an organization.
- Variables:
username (str) – Username of the user
password (str) – Password of the user
firstname (str) – First name of the user
lastname (str) – Last name of the user
email (str) – Email address of the user
organization_id (int) – Foreign key to the organization to which the user belongs
failed_login_attempts (int) – Number of failed login attempts
last_login_attempt (datetime.datetime) – Date and time of the last login attempt
otp_secret (str) – Secret key for one time passwords
organization (
Organization
) – Organization to which the user belongsroles (list[
Role
]) – Roles that the user hasrules (list[
Rule
]) – Rules that the user hascreated_tasks (list[
Task
]) – Tasks that the user has created
- can(resource, scope, operation)#
Check if user is allowed to execute a certain action
- Parameters:
- Returns:
Whether or not user is allowed to execute the requested operation on the resource
- Return type:
bool
- check_password(pw)#
Check if the password is correct
- Parameters:
pw (str) – Password to check
- Returns:
Whether or not the password is correct
- Return type:
bool
- classmethod exists(field, value)#
Checks if user with certain key-value exists
- Parameters:
field (str) – Name of the attribute to check
value (str) – Value of the attribute to check
- Returns:
Whether or not user with given key-value exists
- Return type:
bool
- classmethod get_by_email(email)#
Get a user by their email
- Parameters:
email (str) – Email of the user
- Returns:
User with the given email
- Return type:
- Raises:
NoResultFound – If no user with the given email exists
- classmethod get_by_username(username)#
Get a user by their username
- Parameters:
username (str) – Username of the user
- Returns:
User with the given username
- Return type:
- Raises:
NoResultFound – If no user with the given username exists
- is_blocked(max_failed_attempts, inactivation_in_minutes)#
Check if user can login or if they are temporarily blocked because they entered a wrong password too often
- Parameters:
max_failed_attempts (int) – Maximum number of attempts to login before temporary deactivation
inactivation_minutes (int) – How many minutes an account is deactivated
- Return type:
Tuple
[bool
,Optional
[str
]]- Returns:
bool – Whether or not user is blocked temporarily
str | None – Message if user is blocked, else None
- set_password(pw)#
Set the password of the current user. This function doesn’t save the new password to the database
- Parameters:
pw (str) – The new password
- Returns:
If the new password fails to pass the checks, a message is returned. Else, none is returned
- Return type:
str | None
- classmethod username_exists(username)#
Checks if user with certain username exists
- Parameters:
username (str) – Username to check
- Returns:
Whether or not user with given username exists
- Return type:
bool
Database models that link resources together#
The Member table is used to link organizations and collaborations together. Each line in the table represents that a certain organization is member of a certain collaboration by storing the ids of the organization and collaboration.
- Member#
alias of Table(‘Member’, MetaData(), Column(‘organization_id’, Integer(), ForeignKey(‘organization.id’), table=<Member>), Column(‘collaboration_id’, Integer(), ForeignKey(‘collaboration.id’), table=<Member>), schema=None)
The Permission table defines which roles have been assigned to which users. It can contain multiple entries for the same user if they have been assigned multiple roles.
The UserPermission table defines which extra rules have been assigned to which users. Apart from roles, users may be assigned extra permissions that allow them to execute one specific action. This table is used to store those, and may contain multiple entries for the same user.
- Permission#
alias of Table(‘Permission’, MetaData(), Column(‘role_id’, Integer(), ForeignKey(‘role.id’), table=<Permission>), Column(‘user_id’, Integer(), ForeignKey(‘user.id’), table=<Permission>), schema=None)
- UserPermission#
alias of Table(‘UserPermission’, MetaData(), Column(‘rule_id’, Integer(), ForeignKey(‘rule.id’), table=<UserPermission>), Column(‘user_id’, Integer(), ForeignKey(‘user.id’), table=<UserPermission>), schema=None)
The role_rule_assocation table defines which rules have been assigned to which roles. Each line contains a rule_id that is a member of a certain role_id. Each role will usually have multiple rules assigned to it.
- role_rule_association#
alias of Table(‘role_rule_association’, MetaData(), Column(‘role_id’, Integer(), ForeignKey(‘role.id’), table=<role_rule_association>), Column(‘rule_id’, Integer(), ForeignKey(‘rule.id’), table=<role_rule_association>), schema=None)
5.4.7. Mail service#
- class MailService(app, mail)#
Send emails from the service email account
- Parameters:
app (flask.Flask) – The vantage6 flask application
mail (flask_mail.Mail) – An instance of the Flask mail class
- send_email(subject, sender, recipients, text_body, html_body)#
Send an email.
This is used for service emails, e.g. to help users reset their password.
- Parameters:
subject (str) – Subject of the email
sender (str) – Email address of the sender
recipients (List[str]) – List of email addresses of recipients
text_body (str) – Email body in plain text
html_body (str) – Email body in HTML
- Return type:
None
5.4.8. Default roles#
- get_default_roles(db)#
Get a list containing the default roles and their rules, so that they may be created in the database
- Parameters:
db – The vantage6.server.db module
- Returns:
A list with dictionaries that each describe one of the roles. Each role dictionary contains the following:
- name: str
Name of the role
- description: str
Description of the role
- rules: List[int]
A list of rule id’s that the role contains
- Return type:
List[Dict]