Disclaimer

This document is only collection of author’s notes, experiences and point of views. It is not, in any meaning, either complete description of the topic nor official RTB documentation. It may be inaccurate, incomplete, obsolete, misleading or completely wrong. It may even cause loss of data or damage system integrity. It may not comply with company codex, values, presentation style or economic interests or may reveal company secrets.

As such, it is absolutely prohibited to distribute this document outside of RTB & Co. GmbH. Such an act may result into legal acts against both the sender and the company. But it is allowed to use this document, after review of technical and presentational correctness and accuracy, as an information source for official documentation.

Always contact documentation department for information about current presentation style and allowed formats before creating customer’s documentation.

Integration of sale registration to tax office. Currently Croatia only 🏖️

Overview

There is no security implemented in the application. Everything is open and it is up to the system administrator to enforce security (firewall, VPN …​).

If security is later required please make change request and plan additional development. This application is developed as standalone program independent from PDM.control from following reasons (in the first step):

  • 🔓 Security. It is required that the end user give us (RTB) his own private key issued by tax office together with his password. It was not clarified with product manager yet, if the customer trust us so much. There is always chance to misuse customers' data. There is nobody 100% responsible in our company. All administrators share the same password.

  • 🔑 Security. We can not guarantee that our servers are secured on so high level that nobody can misuse / steal customers’ private keys and certificates. Although the customers certificates are stored securely in Java Key Store the password to the key store is written in plain text file. We do not know how to manage user rights on Windows systems to keep this information secret.

  • ⌚ Reliability. Our servers does not guarantee 24/7 availability. RTB has decided to keep reliability low in order to reduce costs. We do not monitor availability and performance of our hosting. There is also nobody who is dedicated to solve a problem during business time. In case of server failure outside of business hour there is even nobody whom to call. If the server would not work all the PDMs in Croatia would not sell. As far as I know there is no plan to make service redundancy. We also do not have know-how how deploy solution to cloud to increase availability.

  • ⚡ Performance. Our hosting sometimes suffers with very long response time. It may let the end customer to wait too long to get parking ticket. In some rare cases is the server response time longer than PDM timer. It may result in sale registered in tax office but without printed ticket. (May be Dragan can handle it).

If we consider all the points above it is advisable to delegate responsibility for 24/7 availability on the customer.

The solution with standalone application has also its own disadvantages.

  • 👨‍🏭 Out-dating: Two applications requires more maintenance. Our update cycle is very slow and it is possible that with two applications it will be much slower.

  • ⌚ Reliability: We have a lot of issues keeping one server running. More servers increase probability of drop out at lease of one of them.

  • 🤼 Compatibility: We have to guarantee that boot applications are compatible during various updates.

In this document are used snippets of configuration in two forms: YAML and properties. Both of them are interchangeable and you may use the one which is better for you. We generally prefer YAML but sometime the properties format creates shorter snippet. Especially if there is only one configuration key.

Certificate import

First is necessary to prepare certificates. There are two types customers' and FINA’s. Proceed to next chapters and prepare both types of key stores.

Private certificates

Private certificates are the certificates which tax office issued for our customers. Is necessary to prepare Key Store (PKCS 12) with all customers' certificates behind of them the sales register will send payments to the tax office. Assume that your key pair got from tax office is stored in file my_company.p12.

Inspect the key store by following command
$ keytool -list -keystore my_company.p12
Enter keystore password: (1)
Keystore type: PKCS12
Keystore provider: SUN

Your keystore contains 1 entry (2)

cn=fiskal 2, Sep 30, 2019, PrivateKeyEntry, (3)
Certificate fingerprint (SHA-256): 9D:F9:E2:84:72:40:8C:70:52:1B:C1:67:15:24:D6:51:51:5C:9D:6E:35:53:71:20:3F:FE:31:74:78:EE:2B:74
1 Enter password which you have got from tax office.
2 There should be just one key of type PrivateKeyEntry.
3 In case of multiple PrivateKeyEntryies, remember the alias of the wanted one. In this case it is "cn=fiskal 2"
Import the PrivateKeyEntry to tax key store file
$ keytool -importkeystore -v -srcstoretype PKCS12 -deststoretype PKCS12 \
-srckeystore my_company.p12 \ (1)
-destkeystore tax_keys.p12 \ (2)
-srcalias "cn=fiskal 2" \ (3)
-destalias "elektrokem" (4)

Importing keystore  to ...
Enter destination keystore password: (5)
Re-enter new password:
Enter source keystore password: (6)
[Storing fiskal.jks]
1 Path to source certificate
2 Path to target key store
3 Source alias (name) of certificate
4 Target alias (name) of certificate. Under this name it will be stored in the key store. This name will be later referenced in configuration.
5 Password of key store. May be displayed just once if password store already exist.
6 Enter password which you have got from tax office. This process has to be repeated as many times as many key pairs (companies) needs to be added.

Repeat process for all customers' certificates.

Fina certificates

Manual build from source certificates

It seams that FINA’s certificates, at least for test environment, are not trusted by default. In order to use them a FINA key has to be added among trusted certificates in new key store. This key store (let call it FinaDemoRootCA.crt) does not need to be protected because it contains only publicly available information.

If you need to trust all Java trusted certificates and simutaniously FINA, it is recomanded to copy file cacerts.jks from java installation and add certificats there. It is not recommanded to mofify file FinaDemoRootCA.crt directly.

  • It will be overwritten during Java update.

  • Modifying default trust file affects also other applications running on this JVM.

Follow steps below to create JKS with FINA certificate:

Get certificate from FINA.

Documentation says: Procedure of issuing and downloading CIS certificates is available on FINA’s web: http://www.fina.hr/Default.aspx?sec=1541. Details about FINA PKI system are available on FINA’s web pages: http://www.fina.hr/Default.aspx?sec=1799. Assume that a file FinaDemoRootCA.crt has to be downloaded. Import the certificate into new key store cacerts.jks like this.

keytool -import -trustcacerts -keystore cacerts.jks -alias "Fina Demo Root CA" -file FinaDemoRootCA.crt
Owner: CN=Fina Demo Root CA, O=Financijska agencija, C=HR
Issuer: CN=Fina Demo Root CA, O=Financijska agencija, C=HR
Serial number: db5f7ce10000000053281c9f
Valid from: Tue Mar 18 10:45:00 CET 2014 until: Sat Mar 18 11:15:00 CET 2034
Certificate fingerprints:
         SHA1: CF:06:2E:51:85:79:C3:AD:C6:CE:20:A9:4A:88:52:89:88:3B:AA:2A
         SHA256: A0:62:8E:66:BC:FC:6D:ED:7B:6D:84:57:DF:57:AD:54:77:EE:55:B7:D6:E1:16:1E:25:21:2B:D5:B1:B1:E1:4D (1)
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 4096-bit RSA key
Version: 3

Extensions:
  <abbreviated>

Trust this certificate? [no]: (2)
1 Verify fingerprint of the certificate.
2 Type yes if you trust it.

Automatic build (Jenkins)

Trust store which includes all Java trusted certificates and Fina certificates is build by Jenkins. You can download it and directly use. Password is rtbrtb. It is also used by our hosted sale registerers.

Certificate build script do not verify fingerprint of certificate. If you do not trust the certificate from link above, do not use neither this pre-build key store not sale registers managed by RTB.

Installation

Linux

Configuration

Installation procedure is similar to PDM.control. Please refer there (or call our helpdesk) to find out more. The most important steps are repeated here.

Assume that sale register will be installed to path /opt/sreg and it will be run under user pdm in group rtb. Create directory and became user wich will run the application.

$ sudo mkdir /opt/sreg
$ sudo chown pdm:rtb /opt/sreg
$ cd /opt/sreg
$ sudo su pdm

Put here:

  • Application: sale_register.war

  • Application configuration: instance.cfg.yml.

  • Trusted store: cacerts.jks

Adjust instance configuration in file instance.cfg.yml according description in chapter instance configuration. When your are done with this step, come back to continue.

Create directory for project configuration e.g. config

$ mkdir config

Put following files in config directory here:

Continue to sections Project and Country specific settings and follow steps there. When you are done come back here.

Now a you can start sale register by command (assuming you are in /opt/sreg directory:

$ java \
-Djavax.net.ssl.trustStore=cacerts.jks \
-Djavax.net.ssl.trustStoreType=JKS \
-Djavax.net.ssl.trustStorePassword=rtbrtb \
-Dspring.config.location=classpath:/application.yml,file:instance.cfg.yml,file:config/project.cfg.yml \
-jar sale_register.war

Set required permissions and security and test again.

If you do not work with system journal, you can redirect logging to files See Logging section.

Service unit

To start sale register as system service, a systemd configuration file is required. Steps how to create it are the same as for PDM.control.

An example configuration stored in file /etc/systemd/system/pdm_sale_register.service can be used.

Systed unit for sale register
[Unit]
Description=PDM.sale_register

[Service]
WorkingDirectory=/opt/sreg
ExecStart=java \
-Xms50m -Xmx100m \
-Djavax.net.ssl.trustStore=cacerts.jks \
-Djavax.net.ssl.trustStoreType=JKS \
-Djavax.net.ssl.trustStorePassword=rtbrtb \
-Dspring.config.location=classpath:/application.yml,file:instance.cfg.yml,file:config/project.cfg.yml \
-jar sale_register.war
User=pdm
Type=simple
Restart=on-failure
RestartSec=60

[Install]
WantedBy=multi-user.target

Reload deamons and start the service.

$ sudo systemctl daemon-reload
$ sudo systemctl start pdm_sale_register.service

Windows

Installation procedure for windows is very similar to Linux. It requires the same tool set as PDM.control. It means Java, Tomcat (database is not required). Please refer to PDM.control installation manual or call our helpdesk to install it. If explanation above is not enough please contact our help desk.

Server cluster configuration

PDM support connection to secondary sale register, in case that the firs one does not work. This require to keep two servers running with the same configuration. Therefore it is advisable to split configuration into multiple files:

  • Instance specific: instance.cfg.yml

  • Project specific: project.cfg.yml

When configuration is changed on one server, it is easy to synchronize it with the other just by copying files and restart.

Instance

Settings in this file should affect only this instance of sale register. There is nothing to prevent us to put also project configuration. nevertheless if you run multiple servers, it is better to separate them.

Please remember that the configuration is read only during startup. After each configuration change you need to restart service:

  • Linux: sudo systemctl restart sale_register.

  • Windows: Services → appropriate tomcat → restart.

Configuration is done by following file:

server:
  port: 8081 (1)
sale-register:
  pdm:
    connection:
      tcp-port: 8709 (2)
  sellingPointsDir: config (3)
1 Port where web server listens. Currently is web server used only for server monitoring. Nevertheless it is necessary to assign port number which does not collide with other application even if you do not use it.
2 Port where sale register listens. It has to be accessible from outside.
3 Directory where selling points definition (*.csv) is stored. In our case it is project configuration directory config.

A server monitor can be also configured here. See section Server monitor.

Project

Project configuration is mainly composed from

First it is necessary to define which Area belongs to which company and by which sale register service it will be processed. See example below.

Companies mapping
sale-register:
  key-store:
    path: config/tax_keys.p12 (1)
    password: <secret> (2)
  areas:
    1: (3)
      tax-office-ref: hr-test (4)
      company:
        id: 38411868043 (5)
        cert-name: elektrokem (6)
    58:
      tax-office-ref: hr-real
      company:
        id: 38411868043
        cert-name: elektrokem
1 File, where customers' keys are located.
2 Password for key store.
3 Area number.
4 Name of tax register service to register with. Currently: hr-test, hr-real.
5 Company’s ID (JIR in case of Croatia); assigned by tax office.
6 Customer’s certificate alias, as it has been registered in file tax_keys.p12. See section Private certificates.

Country specific settings

Croatia

Croatian tax law requires that every PDM has its own

  • Electronic device identification (EDID).

  • Business premises designation number (BPDN)

numbers. Those are assigned by tax office and can not be derived directly from PDM number or area.

Therefore an external configuration has to be provided in order to create mapping between our PDM identification and tax office identification. The mapping is stored in external CSV files and not in instance configuration, in order to be able to quickly synchronize settings between multiple servers. This CSV files are always the same for all servers, they do not contain any individual configuration.

Nevertheless it is necessary to provide the path to this file in instance configuration file in property sale-register.selling-points-dir.

Example of path definition
sale-register.selling-points-dir=/opt/sreg/config

Format of definition file is Microsoft CSV with headers. Yes, we all love Excel. However as you probably know, value separator generated by Excel is locale dependent. Therefore in your region it may be different then in Germany. Because the program has to take some separator setting we assume that we all live in Germany. It means, that separators are: value ; and string ". If you excel does not generate this, please use LibreOffice, adjust regional settings or replace delimiters in text editor.

Please note that header line with column names must be always provided. The order of column does not matter.

Following column names are expected. It is expected to get value in each column for every row.

  • pdm - PDM number

  • edid - Electronic device identification

  • bpdn - Business premises designation number

Example of CSV configuration file
pdm;edid;bpdn
1;231637843;108
2;231637844;109

Area is not mentioned in the file content. There must be a separate file for each area. The reason is to avoid complex merges, when the administrator gets different configurations from different customers.

File name is derived Area and country. It has format <area>.<country>.csv In case of Croatian and Area 27 the file name will be 27.hr.csv

Logging

By default the application logs to standard out (console). If you want to keep logs in files put following configuration to instance.cfg.yml file.

logging:
  pattern:
    console: (1)
  file:
    name: logs/main.log (2)
    max-history: 30 (3)
    max-size: 100MB (4)
1 Disable console (standard out) logging. Leave content of console: empty.
2 Log file name (for instance, main.log). Names can be an exact location or relative to the current directory.
3 Maximum number of days archive log files are kept.
4 Maximum log file size.

Security

Some file contain sensitive information; especially passwords. It is recommended to protect them on OS level. We use following settings:

$ chmod 600 instance.cfg.yml (1)
$ chmod 600 config/tax_keys.p12 (2)
$ chmod 600 config/project.cfg.yml (3)
1 (May) contain password for monitoring server
2 Private keys of customers
3 Password for tax_keys.p12

Sensitive files can be accessed only by user under which runs sale register process.

Server synchronization

Sale register runs usually on two servers to provide redundancy. Because PDM can request registration from any of those two servers, it is important to keep their configuration in sync. Because we do not want to make every change twice, we change only configuration on primary server and let a script from replica server to synchronize it.

Prerequisites

  1. ssh from replica to primary works (how to).

  2. User that runs the script, can restart sale_register service. We use the same user (pdm) under sale_register service normally run. Add following line to visudo to achieve it.

    sudousers
    #Allow user `pdm` to stop and start sale register service.
    pdm ALL=(ALL) NOPASSWD: /usr/bin/systemctl stop pdm_sale_register,/usr/bin/systemctl start pdm_sale_register
Synchronization script

Put following script to directory /opt/sync_sreg.sh

#!/bin/bash
echo Stopping sale register service ...
sudo systemctl stop pdm_sale_register

echo Copying configuration from primary server ...
rsync -a -P --delete --exclude instance.cfg.yml --exclude logs pdm@primary_server:/opt/sreg/ /opt/sreg/ (1)

echo Starting pdm_sale_register service ...
sudo systemctl start pdm_sale_register
1 Set correct URL of your primary server.
Execution

Run this script under user with required permissions. It will

  • stop sale register service

  • backup old configuration

  • copy configuration from primary with permissions

  • start sale register service

# su pdm
$ cd /opt
$ ./sync_sreg.sh

Remote monitoring

This application supports remote monitoring via Prometheus and SpringBootAdmin. To activate it it is necessary to configure following.

The feature is currently not intended for our customers. It is only for us (RTB) to evaluate it.

Server monitor

Server monitor is optional feature which increases reliability remote server monitoring from central place.

User accounts

In order to access this instance remotely it is necessary to configure security. In our case it means create user with strong password which can access this instance.

Let call this user monitor. The name is not important. Important is, that he/she/it has role MONITOR. Example configuration is below.

Monitoring user
sale-register:
  users:
    -
      name: monitor
      password: <password> (1)
      roles:
        - MONITOR
1 Put user password here.
Monitor configuration

PDM server monitor allows quickly monitor health and availability of servers which supports it. It is required that PDM server monitor server can access the web interface of the application. There may be required a firewall configuration or port forwarding or proxy.

Configuration look like this. All meta data are optional.

Monitoring configuration (example)
spring:
  boot:
    admin:
      client:
        enabled: true
        # User which has rights to read applicaion's actuators.
        user: monitor
        # Password configured in paragraph above.
        password: <monitror's secret password>
        # URL of server monitor.
        url: https://monitor.my-company.com
        instance:
          # URL where monitored application is accesible.
          service-url: https://sreg.my-company.com
          # All options below are optional. See original documentation:
          # https://codecentric.github.io/spring-boot-admin/current/#spring-boot-admin-client
          # Information about company which uses the application
          metadata:
            # Full name of company
            company: RTB GMBH & CO. KG
            contact:
              # Contact phone to IT responsible
              phone: +49 5252 97060
              # Address
              address: Schulze-Delitzsch-Weg 10, 33175 Bad Lippspringe
              # Contact email to IT responsible
              email: it@my-company.com
            # Properties of the instance (not required)
            tags:
              # This is production instance
              production: true
              # Abbreviation of company name
              server: sreg1

Prometheus (experimental)

The application exposes except of standard JVM probes also custom ones. All of them starts with sreg prefix.

It is required that Prometheus server can access the web interface of the application. There may be required a firewall configuration or port forwarding.