Backend documentation¶

Here is the documentation for Backend code. The documentation is not complete, and is automatically generated using the sphinx python package, meaning there can be weird formatting in places, since not all documentation is strictly adhering to a specfic documentation style. To run backend after doing first time initialization, you simply run (from root folder) the below command

python -m src.backend.main

This will spawn an instance of src.backend.mqttclient using the configurations of the project.

For first time initialization, you want to place a metadata.xlsx file in the root location that is configured according to the attached metadata_example.xlsx. Once that is in place, you can initalize backend as so:

python -m src.backend.main -i
[...] Do you wish to convert metadata from an excel (xlsx) file? [y/n]: y
Please input ip-address: 127.0.0.1
and port number: 1883
username: mqttuser
Password: mqttpassword

You will be prompted to fill in mqtt broker information. After the information has been filled out src.backend.main will attempt to spawn src.backend.mqttclient as a subprocess. Whenever it crashes it will attempt to respawn the process. That means that if the mqqt-client is unable to connect to the broker, it will keep respawning the client. If this is the case, kill the process, and then run python -m backend.src.main again with the -i flag, or both the -i and -r flag. You will be prompted to fill out the MQTT broker configuration again. Alternatively, you can edit the MQTT configuration directly inside src/backend/.config/config.toml.

To see all arguments that src.backend.main supports, run

python -m src.backend.main -h

Again, make sure to have an excel file in the root location that is named metadata.xlsx. Otherwise, the init cannot convert the project metadata to its own internal metadata files. Without them, frontend cannot work, and features of backend will not work either, such as raw data conversion and positioning.

Top-level modules¶

main¶

src.backend.main.init_logging()¶

src.backend.main.main() → NoReturn¶: A function that spawns the main MQTT client as a subprocess. If the MQTT client terminates for some reason, this function will restart the client.

mqttclient¶

class src.backend.mqttclient.CustomFormatter(fmt=None, datefmt=None, style='%')¶

Logging Formatter to have custom format for the different logging levels.

FORMATS = {20: '{module:11s} - {message}', 'DEFAULT': '[{asctime}] - {name} - [{levelname:8s}] - {message}'}¶

format(record)¶

Format the specified record as text.

The record’s attribute dictionary is used as the operand to a string formatting operation which yields the returned string. Before formatting the dictionary, a couple of preparatory steps are carried out. The message attribute of the record is computed using LogRecord.getMessage(). If the formatting string uses the time (as determined by a call to usesTime(), formatTime() is called to format the event time. If there is exception information, it is formatted using formatException() and appended to the message.

src.backend.mqttclient.init_logging() → None¶

src.backend.mqttclient.load_iof_mqtt_topics() → List[Tuple[str, int]]¶

src.backend.mqttclient.main() → NoReturn¶

src.backend.mqttclient.mqtt_client_config() → Tuple[str, int, str, str]¶

src.backend.mqttclient.on_connect(mqttc, obj, flags, rc) → None¶

src.backend.mqttclient.on_disconnect(mqttc, obj, rc) → None¶

src.backend.mqttclient.on_log(mqttc, obj, level, string) → None¶

src.backend.mqttclient.on_message(mqttc, obj, msg) → None¶

src.backend.mqttclient.on_publish(mqttmessage_handlerc, obj, mid) → None¶

src.backend.mqttclient.on_subscribe(mqttc, obj, mid, granted_qos) → None¶

src.backend.mqttclient.unpack_json_payload(payload: bytes) → Optional[Mapping[str, Union[str, float]]]¶

initbackend¶

src.backend.initbackend.define_mqtt_config(client: bool = False) → None¶: prompts user to input ip-address, port, username and password for mqtt by calling _mqtt_config_ip_port_usr_pwd(). Writes result to ‘src/backend/.config/config.toml’

src.backend.initbackend.init_iof(args: List[str]) → None¶: inits databases. If args contain –metadata it will also load in metadata if they exist in ‘src/backend/.config/metadata.toml’. If args contain -db argument, database names in ‘dbmanager/databases/’ will be named with arguments passed in -db.

src.backend.initbackend.init_metadata()¶

src.backend.initbackend.iof_ready() → bool¶: Returns True if ready to run iof backend. Else returns False

src.backend.initbackend.reset_iof() → None¶

Message handling¶

conversion¶

src.backend.msghandler.conversion.convert_packet_payload(packet: Dict[str, Union[int, str, float]]) → Dict[str, Union[int, str, float]]¶

Converts payload datafields as suitable.

Iterates through packet datafield and checks whether they are in the ConverisonMapping dict of this module. If a datafield exists, the relevant conversion function is called. The functions will either return converted data, or the same raw data, but with an added or converted datafield in the packet data structure. Datafields not in the conversionMapping are ignored

Parameters: packet – Dict[str, Union[str, int, float]].
Returns: The same packet with modified datafields

msghandler¶

class src.backend.msghandler.msghandler.Message(header: Dict[str, Union[int, str, float]], payload: List[Dict[str, Union[int, str, float]]])¶

exception src.backend.msghandler.msghandler.MessageHandlingError¶

src.backend.msghandler.msghandler.handle_message(data: bytes) → src.backend.msghandler.msghandler.Message¶: Return unpacked and converted internetoffish message as a dictonary with msg header and payload as list data. Adds tbr_serial_id and correct timestamp to each packet, and sets gps data as a packet in payload rather than header. Takes bytes data as input, formatted according to protocol used in SLIM hardware. Each packet is handled so that data is converted or changed appropriately (commCode 3 becomes ‘S256’ f. ex. and tag_data with conversion factor is handled etc.)

packet¶

src.backend.msghandler.packet.get_packet_length_type_and_format(code: int) → Tuple[int, str, List[src.backend.msghandler.protocol.PacketSlice]]¶: Returns packet length and unpacking format based on code. If code is 255, packet is a TBR sensor msg. If code is a (valid) key in msghandler.protocol codetypes dictionary, get associated communication protocol and insert unpacking format for this communication protocol in packet unpacking format. Unpacking formats defined in msghandler.protocol.

src.backend.msghandler.packet.mask(data, lenData, bits, MSB)¶

src.backend.msghandler.packet.unpack_packet(packetData: bytes, format: List[src.backend.msghandler.protocol.PacketSlice]) → Mapping[str, int]¶: Unpacks bytes msg packet using format defined in msghandler.protocol. Iterates trough each segment of a message and unpacks the datafields contained inside. A segment consists of complete byte datafields and/or datafields of bits. Returns packet as a dictionary.

protocol¶

class src.backend.msghandler.protocol.DataField(name: str, length: int, bits: Union[int, NoneType] = None, MSB: Union[bool, NoneType] = None)¶

MSB = None¶

bits = None¶

class src.backend.msghandler.protocol.PacketSlice(numBytes: int, datafields: List[src.backend.msghandler.protocol.DataField])¶

class src.backend.msghandler.protocol.PacketType(type: str, format: List[src.backend.msghandler.protocol.PacketSlice])¶

Databasemanager¶

dbformat¶

src.backend.dbmanager.dbformat.sql_query_gps_create_table_dummy() → Tuple[str, str, Tuple[Union[int, str]]]¶

returns table gps create statement, dummy data insertion, and dummy data. The dummy data is not constructed correctly, all are equal to -1 for clarity. Message_id = -1 also ensures that first message_id will be set to 0.

Example where the right types of data is used in gps message:

(1, 754605000, 33, “low battery”, 7.31442, 13.59674, 1.2, “3D-fix”, 22)

DECIMAL(7, 5) and DECIMAL(8, 5) used for latitude and longitude is not supported in sqlite like other sql solutions. However, the data affinity will be NUMERIC, and the keyword informs users that these numbers should have fixed-precision set to 5 decimal places.

src.backend.dbmanager.dbformat.sql_query_metadata_create_table() → str¶: Returns table metadata create statement. See src/backend/.config/metadata.toml for relevant construction/format of metadata.

src.backend.dbmanager.dbformat.sql_query_tag_create_table_dummy() → Tuple[str, str, Tuple[Union[int, str]]]¶

returns table tage create statement, dummy data insertion, and dummy data. The dummy data is not constructed correctly, all are equal to -1 for clarity. Message_id = -1 also ensures that first message_id will be set to 0.

Example where the right types of data is used in gps message:: (1, 33, 754605000, “S256”, 69, 88, 6.2, 31, 17, 330)

src.backend.dbmanager.dbformat.sql_query_tbr_create_table_dummy() → Tuple[str, str, Tuple[Union[int, str]]]¶

returns table tbr create statement, dummy data insertion, and dummy data. The dummy data is not constructed correctly, all are equal to -1 for clarity. Message_id = -1 also ensures that first message_id will be set to 0.

Example where the right types of data is used in gps message:

(1, 33, 754605000, 9.2, 142, 31, 55, 69, “this is a comment”)

DECIMAL(3, 1) used for temperature is not supported in sqlite. However, the data affinity will be NUMERIC, and the keyword informs users that these numbers should have fixed-precision set to 1 decimal places.

dbinit¶

src.backend.dbmanager.dbinit.check_if_reset_of_iof_wanted(dbName: str, dbBackupName: str) → bool¶: Checks if new database names match old ones, if they do, prompts user to delete old databases. Also checks if old dbDict has different names for databases and prompts user to delete these as well if so.

src.backend.dbmanager.dbinit.databases_ready() → bool¶: Checks if both main and bakup database exists, and return True if they do

src.backend.dbmanager.dbinit.init_databases(dbName: str, dbBackupName: str) → None¶: Creates main_database and backup_database in ‘dbmanager/databases/’. dbName is name of main_database, dbBackupName is name of backup_database. The names will also be saved to ‘src/backend/.config/db_names.toml’.

src.backend.dbmanager.dbinit.reset_databases() → None¶: resets package by deleting databases in ‘dbmanager/datbases/’ and by resetting ‘src/backend/.config/db_names.toml’. Should be called when –reset argument called with main.py

dbmanager¶

msgbackup¶

src.backend.dbmanager.msgbackup.store_message_to_backup_db(msg: Mapping[str, Union[str, float]], msgID: Optional[int] = None) → None¶: Stores raw bytearray message as hex in backup database

msgconversion¶

class src.backend.dbmanager.msgconversion.DatabasePacket(table: str, columns: Tuple[str], values: Tuple[Union[int, str, float]], numOfValues: int = 0, sql_columns: str = '', sql_values: str = '')¶

positioning¶

class src.backend.dbmanager.positioning.CageCircle(center: src.backend.dbmanager.tdoa.Point, radius: float)¶

class src.backend.dbmanager.positioning.CageGeometry(circle: 'CageCircle', lat_A: 'float', lat_B: 'float', lat_C: 'float', lon_A: 'float', lon_B: 'float', lon_C: 'float')¶

class src.backend.dbmanager.positioning.CageMeta(cageName: 'str', tbr: 'List[ListTBR]', depth: 'float', geometry: 'Optional[CageGeometry]')¶

class src.backend.dbmanager.positioning.Point(x: float, y: float)¶

class src.backend.dbmanager.positioning.Position(timestamp: 'int', tag_id: 'int', frequency: 'int', cage_name: 'str', millisecond: 'int', x: 'float', y: 'float', z: 'float', latitude: 'float', longitude: 'float')¶

class src.backend.dbmanager.positioning.TagsMeta(tag_id: 'int', frequency: 'int', cageName: 'str')¶

src.backend.dbmanager.positioning.circleFromThreePoints(P0: src.backend.dbmanager.positioning.Point, P1: src.backend.dbmanager.positioning.Point, P2: src.backend.dbmanager.positioning.Point) → src.backend.dbmanager.positioning.CageCircle¶

Finds and returns minimal circle (center, radius) based on three xy-points.

Function derived from: | https://www.xarg.org/2018/02/create-a-circle-out-of-three-points/

Parameters

P0 – Point 0
P1 – Point 1
P2 – Point 2

Returns

Instance of dataclass ‘Circle’ with center and radius attributes.

src.backend.dbmanager.positioning.init_metadata(old=False) → Optional[Tuple[Dict[str, List[List[int]]], List[src.backend.dbmanager.positioning.TagsMeta]]]¶

Loads positoning metadata from .toml file if it exists.

Loads metadata from toml-file into dictonary, and if successful, iterates through said metadata. For cages metadata, it creates an instance of CageMeta for each cage, and for tags metdata, it creates an instance of TagsMeta for each depth tag. These instances are added to lists ‘cages’ and ‘depth_tags’.

Returns

Returns a tuple of two lists, where the first list contains multiple instances of CageMeta, and the second contains multiple instances of TagsMeta. If metadata cannot be loaded for some reason, the function returns None.

Raises

FileNotFoundError – NB! No metadata position config file found. Can’t do positioning.
Exception – Caught an error while loading metadata positioning

src.backend.dbmanager.positioning.position_database(dbObj: Any)¶

Searches through complete database and returns all found positions as a list.

Goes through all valid tag_id/frequency combinations for a given project and find triplets of them in the database, and uses this as well as station data to position all tag messages from all messages. Returns a list of found positions.

Parameters: dbObj – dbmanager.DatabaseManager instance, with connection to main database.
Returns: List of all found positions, where each position is an instance of dataclass ‘Position’.

src.backend.dbmanager.positioning.position_new_msg(msg: msghandler.Message, dbObj: DatabaseManager) → Optional[List[tdoa.CoordXYZ]]¶

Return list of xyz-position of new msg if triplets of tag detections exists.

Iterates through msg and checks whether two other messages from the same tag_id with matching cage TBR IDs exists in database. If so, runs positoning algorithm on tag detection triplet, and returns a list of the positions.

Parameters

msg – Dictionary following msghandler.Message format, where a string is used for each key, with corresponding values being of type Union[int, str, float].
dbObj – dbmanager.DatabaseManager instance, with open connection to database.

Returns

If any positions has been found for tags in message, returns a list of the positions. If no positions has been found, returns None.

src.backend.dbmanager.positioning.position_tag(cage: src.backend.dbmanager.positioning.CageMeta, stations: src.backend.dbmanager.tdoa.StationData, tag_depth: float, tstamps: src.backend.dbmanager.tdoa.Timestamps, cageVerify: bool = True) → Optional[src.backend.dbmanager.tdoa.CoordXYZ]¶

Attemtps to position and resolve tag positions, returns CoordXYZ if successful.

Uses station data, cage information, and tag data to call TDOA hyperbola based positioning. If successful, attempts to resolve ambiguity of position candidates by viewing distance from stations to candidates, and order of arrival. In addition, if enabled, tries to validate position based on distance from cage center. Returns CoordXYZ(x, y, z) of tag if successful, else returns None.

Parameters

cage – Instance of CageMeta, used to filter based on cage center.
stations – Instance of StationData, containing positions of stations, depth of stations etc. Used in TDOA algorithm, and for arrival-resolvement.
tag_depth – depth of tag. Stations depth is subtracted from tag_depth in TDOA.
tstamps – Instance of dataclass Timestamps, containing second and millisecond values for arrival of message in station A, B and C.
cageVerify – Boolean argument to enable validation based on distance from cage center. By default True.

Returns

If valid position candidates are found from TDOA algorithm, resolvement and validation is performed. If position is successfully found, it is returned, otherwise None is returned.

tdoa¶

class src.backend.dbmanager.tdoa.Circle(center: src.backend.dbmanager.tdoa.Point, radius: float)¶

class src.backend.dbmanager.tdoa.CoordUTM(easting: float, northing: float)¶

class src.backend.dbmanager.tdoa.CoordXYZ(x: float, y: float, z: float, unit: str = 'm')¶

class src.backend.dbmanager.tdoa.LatLong(lat: float, lon: float)¶

class src.backend.dbmanager.tdoa.Point(x: float, y: float)¶

class src.backend.dbmanager.tdoa.StationData(TBR_A: int, TBR_B: int, TBR_C: int, pos_A: src.backend.dbmanager.tdoa.LatLong, pos_B: src.backend.dbmanager.tdoa.LatLong, pos_C: src.backend.dbmanager.tdoa.LatLong, depth: float, theta: float = 0, utm_zone_num: int = 0, utm_zone_let: str = '', utm_A: src.backend.dbmanager.tdoa.CoordUTM = CoordUTM(easting=0, northing=0), utm_B: src.backend.dbmanager.tdoa.CoordUTM = CoordUTM(easting=0, northing=0), utm_C: src.backend.dbmanager.tdoa.CoordUTM = CoordUTM(easting=0, northing=0), xyz_A: src.backend.dbmanager.tdoa.CoordXYZ = CoordXYZ(x=0.0, y=0.0, z=0.0, unit='m'), xyz_B: src.backend.dbmanager.tdoa.CoordXYZ = CoordXYZ(x=0.0, y=0.0, z=0.0, unit='m'), xyz_C: src.backend.dbmanager.tdoa.CoordXYZ = CoordXYZ(x=0.0, y=0.0, z=0.0, unit='m'))¶

StationData class used for positioning.

StationData is used to construct and filter data on a valid form to then try and perform positioning based on the stationdata and other data provided. It is also used to convert between UTM, latlong and local XYZ-cartesian coordinate system. And it is used in verification of position candidates (station positions).

TBR_i¶: Integer ID of stations (where i = A, B, C).

pos_i¶: LatLong position of stations (where i = A, B, C).

depth¶: Shared depth of stations

theta¶: Angle to rotate back from local xyz-system to UTM.

utm_zone_num¶: Integer used in conversion from utm to latlong.

utm_zone_let¶: String used in conversion from utm to latlong.

utm_i¶: UTM Northing and Easting of stations (where i = A, B, C).

xyz_i¶: XYZ-coordinates of stations (where i = A, B, C).

class src.backend.dbmanager.tdoa.Timestamps(sec_a: int, sec_b: int, sec_c: int, msec_a: int, msec_b: int, msec_c: int)¶

Timestamps class used for positioning.

Timestamps is used solve TDOA hyperbola positoning, where its attributes represent time of arrival for each station.

sec_i¶: Integer UTC timestamp (seconds) Time of Arrival (where i = A, B, C).

msec_i¶: Integer millisecond Time of Arrival (where i = A, B, C).

src.backend.dbmanager.tdoa.convert_tag_xyz_to_latlong(pos: src.backend.dbmanager.tdoa.CoordXYZ, station_data: src.backend.dbmanager.tdoa.StationData) → src.backend.dbmanager.tdoa.LatLong¶

Simple function to rotate tag xyz back to utm and convert this to latlong.

Parameters

pos – XYZ-position of tag.
station_data – Instance of dataclass StationData, containing attributes: ‘theta’: used to rotate back to utm. ‘utm_A.easting’ & ‘utm_A.northing’: used to translate back to utm. ‘utm_zone_num’ & ‘utm_zone_letter’: used to convert utm to latlong.

Returns

Returns instance of dataclass LatLong with latitude and longitude values.

src.backend.dbmanager.tdoa.distance_xy(P0: Union[src.backend.dbmanager.tdoa.CoordXYZ, src.backend.dbmanager.tdoa.Point], P1: Union[src.backend.dbmanager.tdoa.CoordXYZ, src.backend.dbmanager.tdoa.Point]) → float¶

Finds and returns the distance between two points.

Parameters

P0 – Can be either dataclass Point or CoordXYZ (both have x and y attributes)
P1 – P0: Can be either dataclass Point or CoordXYZ (both have x and y attributes)

Returns

Returns the found distance as a float.

src.backend.dbmanager.tdoa.resolve_position_based_on_order_of_arrival(tstamps: src.backend.dbmanager.tdoa.Timestamps, station_data: src.backend.dbmanager.tdoa.StationData, positions: numpy.array) → Tuple[src.backend.dbmanager.tdoa.CoordXYZ, src.backend.dbmanager.tdoa.CoordXYZ, Optional[src.backend.dbmanager.tdoa.CoordXYZ]]¶

Simple algorithm to resolve ambiguity of position candidates.

Finds and sorts time of arrival and position for each station. Based on this order, and the distance to the candidates, a position candidate can be concluded in most cases.

Parameters

tstamps – Instance of dataclass Timestamps with attributes ‘sec_i’ and ‘msec_i’ (where i = a, b, c)
station_data – Instance of dataclass StationData containing position of stations.
positions – Position candidates found thus far by TDOA hyperbola algorithm.

Returns

Returns a Tuple of the two position candidates, and the position chosen as a solution. If no candidate can be chosen, both candidates are returned and position solution is returned as None.

src.backend.dbmanager.tdoa.tdoa_hyperbola_algorithm(depth: float, tstamps: src.backend.dbmanager.tdoa.Timestamps, station_data: src.backend.dbmanager.tdoa.StationData) → src.backend.dbmanager.tdoa.CoordXYZ¶

Calculates x-y-z coordinates based on station data, depth and timestamps.

Calculates x-y-z coordinates with algorithm from Bertrand T. Fang’s 1989 paper, | –> ‘Simple Solutions for Hyperbolic and Related Position Fixes’. Algorithm is a ‘Time Difference of Arrival’ (TDOA) technique for positoning, where 3 stations (A, B, C) with known positions, are used to measure difference in times of arrival of signal, leading to a hyperbolic solution for position fix. In this case, providing a 2D solution, while the third dimension z is provided by depth argument.

Parameters

depth – Depth of signal, defining z-value of position [m units]
timestamps – Dataclass ‘Timestamps’ with member variables containing second and millisecond time arrival of signal in station A, B and C.
station_data –
Dataclass ‘StationData’ with member variables containing integer id, latitude/longitude position, depth, and x-y-z position for station A, B, and C [m units].

XY position of A is always (0, 0) XY position of B is always (b, 0) XY position of C is always (cx, cy)

Returns

An instance of dataclass ‘CoordXYZ’ with member variables x, y, and z. Coordinates describe 3D-position of signal in relation to coordinate system defined by station data. The position is returned in unit cm.

src.backend.dbmanager.tdoa.verify_position_within_sea_cage(P0: src.backend.dbmanager.tdoa.CoordXYZ, P1: src.backend.dbmanager.tdoa.CoordXYZ, position: Optional[src.backend.dbmanager.tdoa.CoordXYZ], cageCircle: src.backend.dbmanager.tdoa.Circle, rThresh: float = 1.1) → Optional[src.backend.dbmanager.tdoa.CoordXYZ]¶

Simple function that checks if position candidate is within cage cirumreference.

Checks whether solution (position != None), or candidates (position = None), are within cage cirumreference based on distance from cage center. If both canidates are within the cage, choose the one closest to the center.

Parameters

P0 – XYZ-point candidate 0.
P1 – XYZ-point candidate 1.
position – XYZ-point chosen solution (None if none is chosen so far)
cageCircle – Instance of dataclass Circle. Has attributes ‘radius’ and ‘center’.
rThresh – How much more than the cage radius are we accepting (default=1.1)

Returns

If a valid position is chosen, that position is returned (CoordXYZ). If not, None is returned.