Version 20

Disclaimer

This document is only collection of author’s notes, experiences and point of views. It is not, in any meaning, either complete description of the topic nor official RTB documentation. It may be inaccurate, incomplete, obsolete, misleading or completely wrong. It may even cause loss of data or damage system integrity. It may not comply with company codex, values, presentation style or economic interests or may reveal company secrets.

As such, it is absolutely prohibited to distribute this document outside of RTB & Co. GmbH. Such an action may result into legal acts against both the sender and the company. It is only intended, after review of technical and presentational correctness and accuracy, to be used as an information source for official documentation.

Always contact documentation department for information about current presentation style and allowed formats before creating customer’s documentation.

20.0

First run of UI may take long. The caching mechanism has been changed to make it faster. However caches needs to be rebuild. It heavily depends on free RAM. Number of CPU doesn’t affect building speed. Be sure to have at least 3 GB of free RAM before upgrade. More than 4GB RAM have practically no effect. If you do not have so much memory permanently assign it at least temporarily. Expected rebuild time: 4GB: 5min, 3GB: 15min, 1GB 1: hour. End of cache rebuild is indicated by dropping CPU usage back to normal.

To start cache rebuild go to page Logbook→ Payments. The request to load filter content will take very long or even it may end with empty screen or errors. After a while (observe CPU and wait for end of activity) it should load quickly again.

Some configuration parameters has been relocated. Please follow the sections below and update the configuration.

We do not support multiple application on the same tomcat any more. There is few reasons for it:

Reliability. In some environments we experience unpredictable behavior. By isolating one application in one tomcat we gain better control over it.
Security. If one tomcat crashes only one application crashes.
Memory management. It is much easier fine tune one application then multiple. Tomcat self consumes about 15 MB which is reasonable overhead.
Containers. For containers and Linux servers we provide also builds with embedded tomcat. It allows to run single JAR file directly without pre-installed tomcat with its own configuration.

Configuration changes

E-mail

Now there is only one SMTP server for all spring applications on the same tomcat. This make configuration more flexible and easier to maintain.

As you see from example below, the from and replayTo field still belong the application. Settings of SMTP server are global (spring).

Change previous configuration

pcon:
  mail:
    from: 'DEV 💻 (Gmail 🏙) <xxxxx@gmail.com>'
    replyTo: xxxxx@gmail.com
    smtp:
      host: smtp.gmail.com
      port: 587
      username: xxxxx
      password: 'xxxxx'
    properties:
      smtp:
        auth: true
        starttls:
          enable: true

pcon:
  mail:
    from: 'DEV 💻 (Gmail 🏙) <xxxxx@gmail.com>'
    replyTo: xxxxx@gmail.com
spring:
  mail:
    host: smtp.gmail.com
    port: 587
    username: xxxxx
    password: 'xxxxx'
    properties:
      mail:
        smtp:
          auth: true
          starttls:
            enable: true

Data source

We experienced problems on our servers problem with database connection (maybe wrong DB configuration ?). See MOP-212. Therefore I expose now in the configuration whole HikariCP configuration.

Change previous configuration

pcon:
  jdbc:
    url: jdbc:postgresql://localhost:5432/pcon
    username: my_user_name
    password: <secret>

spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/pcon
    username: my_user_name
    password: <secret>

RTB: Server administrator may change the configuration by setting properties in path

spring.datasource.hikari.?=?

List of available properties can be found in documentation.

Customers: No problem has been reported yet. No action is needed.

Logging

Now there is only one logging path for all applications running on the same tomcat.

Change previous configuration

pcon.path.logging=/var/log/rtb/pdm_control

logging.file.path=/var/log/rtb/pdm_control

New

Integration with Brain Behind
User name of the one who has edited update as latest is shown int table [PCON-1582]
Bank notes are not shown on PDM page [PCON-1747]
Hardware report [PCON-1807]
Clearings can be filtered by payment type [PCON-1875]
Status logbook converted to new UI [PCON-1889]
Clearing logbook converted to new UI [PCON-1890]
Various new status messages
Improved speed of browsing logbooks [PCON-1930]
Improved speed deleting of PDM [PCON-1931]
(Some) API documentation <path_to_pdm_control>/swagger-ui.html (E.g. http://localhost:8080/swagger-ui.htm)
New right Server monitoring. Allows remotely observer server health, resources, performance, configuration … Use only with trusted user. Sensitive information may be revealed. Currently it does following:
- Exposes /actuator endpoint to remotely monitor server.
- Adds Server-Timing header to some HTTP response with additional information about server processing. It contains list of tasks which has been done on the server to process the request together with timing. It can be seen in Chrome development tools in card Network → Timing.
Table to visualize disposable bonus codes. Please note that default bonus code validity is now 50 years (previously was 1000 years). [PCON-1953]

Improvements

Format of log files has been changed in order make analysis automation easier. All log files contains timestamps with millisecond precision to allow to join multiple log files together. In Linux server there is also available log from embedded tomcat http.log together with the other files in directory /var/log/rtb/control. Some log files has been renamed
- events-debug.log→ main.log
- events.log → errors.log
- actions.log → user_actions.log
Various performance improvements.

20.1

Improvements

Added real time request 81 (Delete update memory) [PCON-1986].

Changes

Fixed logger default configuration. Temporary line logging.level.de.rtb=trace can be removed.
OCIF types renamed see [PCON-1987].
- OCIF_0: OCIF / Stadtkarte → Bank card
- OCIF_1: OCIF 1 → City bonus
Remote commands are visible only for ROLE_SERVICE [PCON-1983].
Bonus can managed only by role BONUS_ROLE_MANAGE [PCON-1884].
Pager on PDM logs. Be aware of the fact that number of pages is only estimation. Do no make any calculations or assumptions based on this number.

20.2

Fixes

Long processing time on PDM page
- Run-time
- Connections
- Logbooks
Long processing time when new PDM is connected to the PDM.control

There was a problem that in some cases the PDM without history can block server for long time. This is partly resolved by optimizing SQL and partly by limiting history to 30 days.

Changes

History on PDM page is limited to 30 days. It you need more go to reports.
Cash box level is displayed after first clearing.

20.3

New status messages.
Property pcon.mail.from is not required any more. If it is not set, then sender address is used.
Improved logging.

20.4

Multiplication bug

Cause

There is an issue in PDM.control [PCON-2101] which causes, that the same message from PDM may be stored to database multiple times, if PDM repeats it. The issue is present for half a decade bud was not reported so far. There is high probability that every PDM.control (especially if it runs on multiple CPUs) is affected by this issue.

Unfortunately our company could not decide if this problem should be fixed globally or selectively. Therefor those scrips are not part of automatic database migration, which is done during version upgrade. The decision is left on system administrator. Please keep in mind, that deleting historical data will invalidate all previously created reports. Please call our help desk if you have additional questions to this topic.

Steps bellow help to identify if your instance of PDM.control is affected by this issue. Please be prepared that every query present bellow may run several minutes, depending on database size and system performance.

Query depends on database schema and work with version 20.4. They may not work in future releases.

Check

Multiplied database records can by checked by this query.

Multiplication check query

SELECT
-- Payment
(WITH mul_translog AS (
  SELECT tl_pdm_id, tl_tracer, tl_ticket_no, tl_pay_date_time, count(tl_tracer) as cnt
  FROM translog
  GROUP BY tl_pdm_id, tl_ticket_no, tl_pay_date_time, tl_tracer
  HAVING count(tl_tracer) > 1
  ORDER BY count(tl_tracer) desc, tl_pdm_id,  tl_ticket_no, tl_tracer, tl_pay_date_time
)
SELECT sum (cnt - 1) as payment FROM mul_translog),
-- Status
(WITH mul_logbook AS(
  SELECT log_pdm_id, log_tracer, log_msg_nr, log_date_pdm, count(log_tracer) cnt
  FROM logbook
  GROUP BY log_pdm_id, log_msg_nr, log_date_pdm, log_tracer
  HAVING count(log_tracer) > 1
  ORDER BY count(log_tracer) desc, log_pdm_id, log_tracer, log_msg_nr
)
SELECT sum (cnt - 1) as status FROM mul_logbook),
-- Clearing
(WITH mul_clearing as (
  SELECT clr_pdm_id, clr_tracer, count(clr_tracer) AS cnt
  FROM clearing
  GROUP BY clr_pdm_id, clr_date_pdm, clr_tracer
  HAVING count(clr_pdm_id) > 1
  ORDER BY count(clr_tracer) asc, clr_pdm_id, clr_date_pdm, clr_tracer
)
SELECT sum (cnt - 1) as clearing FROM mul_clearing)
;

Table 1. Example result
payment	status	clearing
10731	12254	4334

The number in each column tells how many duplicated records there is for each type. It there is a number greater then 0, then there are duplicated records. They can be deleted by following queries. Each query leave the first appearance of record in the database and deletes consequent ones.

Delete

Payments

Delete multiplied payments

DELETE
FROM translog a
USING translog b
WHERE
  a.tl_id > b.tl_id
  AND a.tl_pdm_id = b.tl_pdm_id
  AND a.tl_tracer = b.tl_tracer
  AND a.tl_ticket_no = b.tl_ticket_no
  AND a.tl_pay_date_time = b.tl_pay_date_time;

Example result

DELETE 10731

Number of deleted records should math number of discovered multiplications.

Status

Status messages may be active (displayed on PDM) therefore they need to be deleted in two steps.

Active statuses

You may (optionally) preview what will be deleted by following query.

Preview active status messages with multiplication

WITH mul_log AS (
  -- Make windows by multiplication criteria
  SELECT
  log_id, log_pdm_id, log_msg_nr, log_date_pdm, log_tracer,
  ROW_NUMBER() OVER w_mul_crit as rownr,
  count(*) OVER w_mul_crit as cnt
  FROM logbook
  WINDOW w_mul_crit AS (PARTITION BY log_pdm_id, log_msg_nr, log_date_pdm, log_tracer ORDER BY log_date_pdm)
)
-- Show only multiplied active (visible on PDM page) statuses
SELECT log_id, log_pdm_id, log_msg_nr, log_date_pdm, log_tracer
FROM mul_log
JOIN pdm_status ON pds_log_id = log_id
WHERE cnt > 1

Table 2. Example of active statuses
log_id	log_pdm_id	log_msg_nr	log_date_pdm	log_tracer
4075493	1309	671	2020-08-31 17:24:00+02	3729
4075239	2134	671	2020-08-31 17:24:00+02	11007
4075579	2159	671	2020-08-31 17:22:00+02	1635
4075180	3093	671	2020-08-31 17:21:00+02	213

Then run query below to delete multiplied instances of those records. If the table above is empty you may skip this step.

Deletion of multiplied copies of active statuses

DO
$$
DECLARE r record;
BEGIN
  FOR r IN
    WITH mul_log AS (
      SELECT
      log_id, log_pdm_id, log_msg_nr, log_date_pdm, log_tracer,
      ROW_NUMBER() OVER w_mul_crit as rownr,
      count(*) OVER w_mul_crit as cnt
      FROM logbook
      WINDOW w_mul_crit AS (PARTITION BY log_pdm_id, log_msg_nr, log_date_pdm, log_tracer)
    )
    SELECT log_id, log_pdm_id, log_msg_nr, log_date_pdm, log_tracer
    FROM mul_log
    JOIN pdm_status ON pds_log_id = log_id
    WHERE cnt > 1
  LOOP
    raise notice 'Deleting record %', r;
    DELETE FROM logbook
    WHERE
    log_id != r.log_id AND
    log_pdm_id = r.log_pdm_id AND
    log_msg_nr=r.log_msg_nr AND
    log_date_pdm =r.log_date_pdm AND
    log_tracer = r.log_tracer;
  END LOOP;
END
$$

Non active statuses

Then delete standard (non active) statuses).

Delete multiplied status

DELETE
FROM logbook as a
USING logbook as b
WHERE
a.log_id > b.log_id
AND a.log_pdm_id = b.log_pdm_id
AND a.log_tracer = b.log_tracer
AND a.log_msg_nr = b.log_msg_nr
AND a.log_date_pdm = b.log_date_pdm

Example output

DELETE 12254

Clearings

Delete multiplied clearings

DELETE
FROM clearing AS a
USING clearing AS b
WHERE
a.clr_id > b.clr_id
AND a.clr_pdm_id = b.clr_pdm_id
AND a.clr_tracer = b.clr_tracer
AND a.clr_date_pdm = b.clr_date_pdm

20.5

Fixes

Fixed incorrect time displayed on Apple devices in reports and remote actions.
Fixed excel export of system messages
FW version is shown in HW report instead of CPU number.
Fixed password change in user profile page.

20.10

This is not bug fix, but new release with new features () and bugs () !

The company decided to quickly release new functions. Nevertheless it is not possible to put them to version 21.

Version 21 is already prepared for the test. There is already PDM.firmware installed on the street, which has features, that start to work when PDM.control 21 is detected as communication partner.

This version has to has number higher then 20.4 and lower then 21.0. I also may need some spare numbers for bug fixes of version 20 (like 20.5, …) Therefore this version is named 20.10.

New

Feratel

We support QR code from feratel. See feature description.

Before configuration in PDM.control is done, be sure, that you have ROLE_BONUS_MANAGE right.

It is necessary to configure and allow the feature in PDM.control. Go to Area Services Feratel and put parameters got from feratel to appropriate fields. There is also some configuration on PDM required. This is probably described in PDM’s release notes.

Other

Secondary task executor

There has been created new task executor for blocking 3^rd party WEB API calls. If real time request can block, then it is executed on secondary task executor. Such a requests do not block PDM.control’s standard processing. It stays responsive even in case that external service does not work or is slow.

This change affects also WES, Brain Behind and Feratel implementation.

Connection timeouts

Please note that standard socket time outs for communication with WEB services has been changed. This change has been introduced after observation in Italy, where some requests took more then 60 seconds, because of issue of our partner.

Currently there are following:

Connection: 2s
Read: 3s

If web service does not answer within this time, it is disconnected. Nobody wants to stay in front of PDM 60 seconds, just to find out that service does not work.

20.11

Bug fixes

Fixed export of payments via api /api/data/payments/byOrder.xml.

20.12

Bug fixes

Fixed integration with Brain Behind.
If user has ROLE_SERVER_MONITOR it is shown reason why paper usage on PDM page cannot be shown.
New status messages.

20.13

Bug fixes

Fixed bonus time formatting in "payments" log book.
Fixed bonus time formatting in "basic bonus" table.
Fixed wrong status on Feratel area service.