All-inclusive OSS Platform for Client/Server Applications

Status

Guidelines

Make it work, make it right, make it fast. -Kent Beck

The only way to go fast, is to go well. -Robert C. Martin

I built the platform to enable smaller developers to rapidly create professional software. The goal is to focus on the needs of business and customers, while being able to rely on a stable technical foundation. This foundation is the platform.

The platform provides everything needed for developers to deliver professional, business centric, web-based, client/server applications. It aims to be powerful and relatively simple, self contained and not reliant on external services, so the applications based on it can run on bare metal and in the cloud, on Linux and Windows.

The platform is not suited for every type of application and scale. It's designed for small and medium sized business applications in the realm of digitalization, automation and optimization of processes. For all business and customer facing components, function and user experience have priority over a custom and fancy UI design.

The platform allows developers to bootstrap a complete development environment. This includes developer machines on Linux or Windows and a development server that provides SCM, CI/CD capabilities and a repository manager.

While the platform contains original pieces of software, it's standing on the giant shoulders of Free/Libre and Open Source Software. Bringing all these tools, libraries and frameworks together in a comprehensive form, I hope the platform can deliver real value for developers and is more than the sum of its parts.

Architecture

The original pieces of software provided are the meta and bootstrap frameworks.

meta is a full backend/frontend solution for building web-based client/server applications in any domain, hence its name.

bootstrap is a tool to build modules for installing and provisioning a system with software.

util is a collection basic Java/JavaScript building blocks used by all parts of the platform and in applications.

Util

Meta

  • meta-core
    • Shared client/server meta model and protocol in Java and JavaScript
    • JavaDoc
    • JSDoc
  • meta-client
    • Client implementations and utilities in Java and JavaScript
    • JavaDoc
    • JSDoc
  • meta-engine
    • Java based server, made up of a collection of services that can be expanded by applications with their own
    • Provides WebSocket and REST API for clients
  • meta-test
  • meta-admin
    • Web application to manage and monitor all functionality offered by meta-engine
  • meta-admin-test

Bootstrap

  • bootstrap-core
    • Java based framework for building, provisioning and executing bootstrap modules on Linux and Windows
  • bootstrap-core-test
  • Bootstrap modules
    • Bootstrap modules are based around simple JSON definitions that contain actions needed to install/configure software
    • Bootstrap modules (eg. bootstrap-dev-vm for installing a developer machine) are in the top level bootstrap module
    • Custom variations and build/provision configs are in a seperate software repository. In case of the platform platform-bootstrap
  • bootstrap-test
    • Automated tests for provisioning bootstrap modules

Source repositories

Fossil SCM repositories:

If you just want to look at the source, download fossil binary and checkout the repositories:

fossil clone https://platform:platform@dev.rswk.ch/scm/platform

For a full walk-through, see the Setup How-To.

Objectives

  • Automate what you can. Almost automated is better than nothing. Use automation as form of documentation
  • Frequently update dependencies. Sometimes it's good to be on the bleeding edge
  • Keep it simple, sometimes boring is better
  • Minimize abstractions, dependencies and vendor lockin
  • Remove cruft, YAGNI
  • There are no solutions, only trade-offs
  • Version numbers are a TL;DR for release notes

Code

  • Use brackets for control flow statements
    if (...) {
        return ...;
    }
    return ...;
    
  • Use brackets for functions/lambdas
    items.stream().filter((item) -> { // Java
        return ...;
    }).findFirst();
    
    items.filter((item) => item.isNew()); // JavaScript
    
    items.filter(function (item) { // JavaScript multiple lines
        return ...;
    });
    
  • Order attributes, fields, parameters, properties, etc. in ASCII order
    public void login(final String password, final String user) ...
    
    enum ContactType {
        EMAIL, MAIL, PHONE
    }
    
    function (password, user) ...
    
    search({limit: 100, name: "Bob"});
    
    <form data-bind="submit: register" id="register" novalidate>
    
  • Don't use preincrement/postincrement operators
    i = i + 1;
    
  • Don't use negation !
    if (hasEnded == false) // Java
    if (hasEnded === false) // JavaScript
    if (Boolean(convertedValue) === false)
    
  • Use < and <= for comparisons
    if (0 < limit && limit < 10) {}
    
  • Use constants first in comparisons
    if (MARKER.equals(currentMarker)) {}
    
  • Use log format with comma separated properties
    LOG.debug("message, property1={}, property2={}", property1, property2); // Java
    LOG.debug(`message, property1=${property1}, property2=${property2}`); // JavaScript
    
    • Lists should be output in format property3=[el1,el2,el3]
  • Comments
    • Put effort into expressive code and use descriptive variable and method names
    • Comments are a last resort
    • Comments lie
    • Use // FIXME and // TODO

Formatting

Setup according to dev-vm guide and use IntelliJ formatter and common sense/existing code.

Documentation

  • Maven Site
    • Contents of src/site
      • Explanations, how-to guides and tutorials
      • Boostrap module reference documentation in bootstrap.md
  • JavaDoc
    • Reference documentation for Java code
    • Packages with package-info.java
      • Use for overview, architecture, etc.
    • Java files according to JavaDoc standards
  • JSDoc
    • Reference documentation for JavaScript code
    • Module documentation (including screenshots for clients) in src/main/resources/web-doc.md
      • Optional JSDoc config in src/main/resources/web-doc-config.json
    • Individual modules with functions and so on according to JSDoc standards

Build

  • All artifacts are built with Maven, or by Maven triggering NPM for JavaScript modules
  • Java JAR artifacts
    • JAR is based on a single Maven module
    • Code is structured into Java packages in src/main/java
    • Module dependencies are handled by Maven in pom.xml
    • Source is compiled and packaged as JAR/Shaded JAR
      • A source JAR is also generated and installed to the Maven repository
    • Test modules are regular Maven modules and seperated from the modules under test
    • JAR files are published to hosted Maven repository on Nexus server
  • JavaScript
    • Code is structured into ES6 modules in src/main/js/modules with additional subdirectories
      • css for CSS
      • static for audio, images and other static files
    • Dependencies to other NPM packages are configured in src/main/resources/js/package.json
    • NPM packages are managed in a workspace using the top level package.json
    • Shell scripts in util-web, used in Maven build profiles
      • web-install
        • Run npm install
      • web-build
        • Run npm run build
      • web-publish
        • Run npm publish
      • web-doc
        • Run npm run doc
    • ZIP web application artifacts
      • Requires web-build.js script
      • Static files are copied using maven-resources-plugin:copy-resources
      • Published to hosted Maven repository on Nexus server using maven-assembly-plugin with web-dist.xml assembly from util-web
      • During development
        • Run npm run build -- watch to start esbuild watch
        • Run mvn resources:copy-resources@web-build-resources to copy static files
    • NPM package artifacts
      • Published to hosted NPM registry on Nexus server using web-publish
  • Bootstrap ZIP artifacts
    • ZIP is based on Maven module
    • Bootstrap resources are in src/main/resources/bootstrap
      • Some standard directories are expected, but all directories and files are packaged into the ZIP
        • config contains server and client configuration. These are usually files that contain variables (eg. {serverDomain}) that are replaced by specific values from bootstrap.json or environment variables during execution
        • definitions contains bootstrap definition JSON files that describe the installation process
        • fixture contains static files that are used as part of the installation, eg. shell scripts, static resources, etc.
    • Dependencies to other bootstrap module are resolved using Maven in pom.xml. The contents of these bootstrap modules become part of the final ZIP, so be aware of file name clashes. All files from these included modules can be referenced like they are part of the current bootstrap module
    • Dependencies on JAR and ZIP binary artifacts (or any other Maven artifact) are resolved by using maven-dependency-plugin. The articacts are copied into the bin directory in the final ZIP
  • Documentation Maven Site
    • Site is configured/enabled on the root Maven module
      • Generated using maven-site-plugin
      • Configuration in reporting section of pom.xml
      • Site generation for sub modules is disabled using <maven.site.skip>true</maven.site.skip>
    • Aggregated JavaDoc is generated using maven-javadoc-plugin
    • JSDoc is generated by web-doc shell script in individual modules and written to src/site/resources/jsdoc of root Maven module

Test

  • In general
    • Unit tests follow the pattern of
      /* Given */
      /* When */
      /* Then */
      
  • JUnit Java tests
    • Unit, integration tests for Java code
    • Test classes are in a test package
      • Eg. for ch.rswk.util.core in ch.rswk.util.core.test
    • Test resources are in src/main/resources/fixture
    • Tests are run during build using maven-surefire-plugin/maven-failsafe-plugin
  • QUnit JavaScript tests
    • Unit tests for JavaScript code
    • Test suite is in src/main/js/qunit.html and test modules are imported and run in src/main/js/modules/tests.js
    • Test module is named after the module under test, eg. src/main/js/modules/client-test.js for client module
    • Tests are run in WebDriver UI tests, see below
  • WebDriver UI tests
  • Bootstrap provisioning tests
    • BootstrapTest classes are in test package and extend AbstractBootstrapTest
    • Tests are not run as part of CI builds, but used as a manual tool to test bootstrap modules
    • Each test case
      • Builds bootstrap module for target environment
      • Creates and provisions new server instance with module
      • Deletes server instance when done

Commit

Checklist:

  • Tests green?
    • Tests for new code?
  • Changed configurations?
    • Adapt bootstrap configurations
  • Documentation updated?
  • Release notes updated?

Commit message format:

Most important thing I did in this commit, a good summary of my changes.

- List of things that changed
- More stuff

Release

  • platform
    • This also applies to repositories based on the platform, like platform-archetype
    • SCM branches
      • trunk
        • Development happens here
        • Always compiles
        • Tests are green
      • stable
        • Manually merge/graft commits from trunk
        • No development, just minor commits (release notes, fixes)
      • Creating more branches is always an option
    • CI jobs
      • platform-snapshot
        • Build, test and deploy contents of trunk to Nexus
      • platform-release
        • Build, test and deploy contents of stable to Nexus
        • Manual steps are required in preparation of and after a release, see Release document
  • platform-bootstrap
    • This also applies to bootstrap repositories based on the platform, like platform-archetype-bootstrap
    • SCM branches
      • trunk
        • Root directory per customer, eg. customerone
        • One root directory for the organization itself, eg. mycompany for infrastructure and development bootstrap modules
        • Root directories contain one directory per bootstrap module, eg. bootstrap-myapp-server
        • Bootstrap module directory contains the Jenkins job configuration, build/provision JSON configs and optional customizing files
    • CI jobs
      • One per bootstrap module, eg. bootstrap-myapp-server
      • Parameters for build (update/install), customer, etc.

Versioning

platform versioning is based on Semantic versioning. Major is not expected to change until the current rewrite. When Minor changes, it's a good idea to check the release notes. Changes in Patch means internal changes, fixes or new features that should be backwards compatible.

Consider using a versioning scheme that fits your development style:

Setup How-To

Requirements:

  • dev-vm developer machine
    • Ubuntu 22.10 or Windows 11
    • 4+ CPU cores
    • 4+ GB RAM
    • 20 GB disk space
  • dev-server development server
    • Ubuntu 22.04.01
    • 4 CPU cores
    • 8 GB RAM
    • 100+ GB disk space (depending on number of stored/archived artifacts)

Bootstrap dev-vm

Setup and bootstrap VM

Setup a fresh VM, eg. using VirtualBox, using one of the ISOs:

  • Ubuntu 23.10
  • Windows 11
    • Download Windows 11 Disk Image (ISO)
    • Installation
      • I don't have a product key > Windows 11 Pro
      • Setup for work
      • Sign-in options > Domain join

Once the OS installation is finished, use the following scripts to start the bootstrap process.

  • Ubuntu terminal
    • Download bootstrap module
      cd /var/tmp
      wget -O bootstrap.zip 'https://rswk.ch/bootstrap-dev-vm-nix.zip'
      
    • Unzip module
      sudo apt install -y unzip
      unzip bootstrap.zip -d bootstrap
      cd bootstrap
      chmod +x bootstrap.sh
      
    • Execute module
      sudo ./bootstrap.sh
      
  • Windows PowerShell/Terminal as Admin
    • Download bootstrap module
      mkdir C:\tmp
      cd C:\tmp
      Set-Variable ProgressPreference SilentlyContinue
      Invoke-WebRequest -Uri 'https://rswk.ch/bootstrap-dev-vm-win.zip' -OutFile bootstrap.zip
      
    • Unzip module
      Expand-Archive -LiteralPath bootstrap.zip -DestinationPath bootstrap
      cd bootstrap
      
    • Execute module
      Set-ExecutionPolicy Unrestricted
      .\bootstrap.ps1
      
  • In case of errors, check bootstrap.log in the bootstrap directory
    ERROR restart point created. see log for errors, fix errors, then run bootstrap script again to resume execution
    
  • Wait for bootstrap process to finish
    executeModule done, elapsed=...
    

Continue with the manual steps to complete the setup.

Explore platform-archetype

Before we dive into the example application, let's make sure everything is setup correctly by running the meta-admin web application:

  • In IDEA, look for module meta-admin-test in framework/meta
    • Open WebDriverTest class
      • In index method, add breakpoint on line first of line after runUiTest
      • Make sure break point suspend option is set to Thread
    • Click on run symbol for index method and select Debug index()
    • Two browser windows should open, with Chromium opening the meta-admin client at https://localhost.platform.rswk.ch:9090/meta-admin/src/main/resources/
    • Login with user plat and password plat
    • TODO do something in admin?
  • Abort the test case

With that done, we can explore the archetype myapp application. It's part of platform-archetyp, the reference for a repository based on platform.

  • Run a full Maven build of platform-archetype with mvn -DskipTests
    • Linux ~/scm/platform-archetype
    • Windows C:\scm\platform-archetype
  • In IDEA, use File > Open and select the directory we used above
    • Note that we have to repeat most of the project setup steps in Project Structure and Settings, like we did for the platform repository
  • Explore myapp-core, myapp-engine, myapp-test modules in application/myapp
  • In Visual Studio Code
    • Explore platform-archetype/application/myapp/myapp-web/src/main/resources for the client application
  • Now everything is ready to try, extend and test the myapp application
  • Debug WebDriverTest#index like we did in meta-admin-test
  • Login to the client application using same login as we did for meta-admin
  • Open the client in a second tab, then click on one of the tiles and the clients should synchronize
  • TODO explain how things work together. model, backend service, client, tests, etc.

In addition to platform-archetyp, there is also a reference bootstrap source repository in platform-archetype-bootstrap containing bootstrap module customizations.

Bootstrap watchdog-server

  • Watchdog is a central meta-engine instance with meta-admin client that is used for monitoring and maintenance for other meta-engine based servers in an organization
  • It is not required for other engines to work, but it is required for automated provisioning of bootstrap modules, either for testing with BootstrapTest or for Jenkins jobs that deploy bootstrap modules
  • Setup Ubuntu VM, eg. with Vultr or any cloud computing provider
  • Setup a DNS A record for domain you control that points to the IP of the VM
  • Example domain used for watchdog server is watchdog.mycompany.ch
    • Replace all instances in platform-archetype-bootstrap of this with a domain you control and would like to use
  • Build bootstrap-watchdog-server module in IDEA
    • In platform project under framework/bootstrap/bootstrap-core, right click on Main.java > Modify Run Configuration
    • Name Main Watchdog
    • Program arguments (On Linux, use ~/scm/ instead of C:/scm)
      -build C:/scm/platform-archetype-bootstrap/mycompany/bootstrap-watchdog-server/build.json
      -customize C:/scm/platform-archetype-bootstrap/mycompany/bootstrap-watchdog-server
      -merge C:/scm/platform-archetype-bootstrap/mycompany/bootstrap-watchdog-server/bootstrap_install.json
      -module C:/scm/platform/bootstrap/bootstrap-watchdog-server
      
    • Environment variables for variables in bootstrap.json
      EMAIL_PASSWORD=my_email_password;
      WATCHDOG_PW=my_watchdog_user_pw;
      

Upload and bootstrap watchdog-server

  • Upload platform/bootstrap/bootstrap-watchdog-server/target/bootstrap-watchdog-server-VERSION.zip to server's /var/tmp directory
  • Unzip module
    sudo apt install -y unzip
    find /var/tmp -name 'bootstrap-*.zip' -exec unzip {} -d /var/tmp/bootstrap \;
    cd /var/tmp/bootstrap
    chmod +x bootstrap.sh
    
  • Execute module
    sudo ./bootstrap.sh
    
  • meta-admin should be available at https://watchdog.mycompany.ch. Login with user watchdog and password mywatchdogpw
  • To have a different user or additional users for the other engines to use, create fixture.json in platform-archetype-bootstrap/mycompany/bootstrap-watchdog-server/config/engine to override the default fixtures in platform/bootstrap/bootstrap-watchdog-server/src/main/resources/bootstrap/config/engine/fixture.json

Bootstrap myapp-server

  • Let's try automated provisioning with BootstrapTest using the watchdog server we just setup
  • BootstrapTest (which uses Provision class under the hood) uses cloud computing provider Vultr to create and deploy VMs. See Vultr Cheatsheet
  • In IDEA, open BootstrapTest in platform-archetype/bootstrap/bootstrap-test
    • Remove @Disabled annotation
    • This test uses localhost.platform.rswk.ch domain in configuration files. To access the server after installation, open your local hosts file and temporarily change the entry for localhost.platform.rswk.ch to the IP of the deployed VM
      • Linux /etc/hosts
      • Windows C:/Windows/System32/drivers/etc/hosts
    • Test can also run with real domains
      • Setup DNS A record for a reserved IP with Vultr
        • Set keystoreFile in propecrties map to AbstractBootstrapTest.CERTBOT_KEYSTORE_FILE
        • Set serverDomain in properties map, eg. to test.mycompany.ch
        • Pass certbot flag to newProvisionTestExecuteConfig method
        • Set provisiontest_vultr_reservedip to the reserved IP ID
  • Right click on BootstrapTest.java > Modify Run Configuration
    • Name BootstrapTest MyAppServer
    • Environment variables
      provisiontest_vultr_apikey=XXX;
      provisiontest_vultr_firewall=XXX;
      provisiontest_vultr_reservedip=;
      provisiontest_vultr_serverpassword=Hu3m4?mFKk2S9!=;
      provisiontest_vultr_sshkey=XXX;
      provisiontest_watchdog_engine=watchdog.mycompany.ch;
      provisiontest_watchdog_password=my_watchdog_user_pw;
      provisiontest_watchdog_user=watchdog;
      
    • All XXX above need to be replaced with actual values from your Vultr settings. See Vultr Cheatsheet
  • Set a breakpoint in myAppServer on assertWatchdogStatus call
  • When breakpoint is reached, the server should be up and running and the client accessible at https://localhost.platform.rswk.ch (or the domain you specified)
    • Check your watchdog server. It should have received the installation status of the new server with a log of the bootstrap process
  • Resume the debugging session and the test resources are cleaned up. If everything went as expected, the test should continue to provision the next target environment

Bootstrap dev-server

For professional software development, it's recommended to setup a development server where all the source repositories, automated CI/CD builds and binary artifacts can be managed/stored.

  • Setup Ubuntu VM, eg. with Vultr, another cloud computing provider, or even a local hypervisor software if you want to run the server on premise
  • Setup a DNS A record for domain you control that points to the IP of the VM
  • Example domain used for development server is dev.mycompany.ch
    • Replace all instances of dev.mycompany.ch in platform-archetype-bootstrap with the domain you control and would like to use
  • To just test the setup of a development server, you can run BootstrapTest#devServer in platform/bootstrap/bootstrap-test, like we did with myapp-server
  • Build bootstrap-dev-server module in IDEA
    • In platform project under framework/bootstrap/bootstrap-core, right click on Main.java > Modify Run Configuration
    • Name Main DevServer
    • Program arguments
      -build C:/scm/platform-archetype-bootstrap/mycompany/bootstrap-dev-server/build.json
      -customize C:/scm/platform-archetype-bootstrap/mycompany/bootstrap-dev-server
      -module C:/scm/platform/bootstrap/bootstrap-dev-server
      
    • Environment variables used in boostrap.json
      NEXUS_PW=my_nexus_admin_password;
      

Upload and bootstrap dev-server

  • Upload platform/bootstrap/bootstrap-dev-server/target/bootstrap-dev-server-VERSION.zip to server's /var/tmp directory
  • Unzip module
    sudo apt install -y unzip
    find /var/tmp -name 'bootstrap-*.zip' -exec unzip {} -d /var/tmp/bootstrap \;
    cd /var/tmp/bootstrap
    chmod +x bootstrap.sh
    
  • Execute module
    sudo ./bootstrap.sh
    

Continue with the manual steps to complete the setup.

Turtles all the way down

  • At this point we have setup
    • Developer machine
    • Development server
    • Watchdog server
  • But all of this was done from the local developer machine. With the development server, we can now store and build everything from a central place and get other people setup as well
  • Clone the mycompany and mycompany-boostrap repositories to your local machine
    • Copy contents of platform-archetype to mycompany and platform-archetype-bootstrap to mycompany-boostrap
    • Rename artifacts, packages, etc. to match your organization
    • Commit changes to server
  • Adapt CI jobs in Jenkins to match your organization
    • Repository and job names
    • bootstrap-dev-vm
      • Build bootstrap module for developer machines based on the customizing in platform-archetype-bootstrap/mycompany/bootstrap-dev-vm
    • bootstrap-dev-server
      • Build bootstrap module for development server based on the customizing in platform-archetype-bootstrap/mycompany/bootstrap-dev-server
    • bootstrap-watchdog-server
      • Build bootstrap module for watchdog server with customizing in platform-archetype-bootstrap/mycompany/bootstrap-watchdog-server
        • Set BUILD param to install to build the module. Must be manually installed on a new watchdog server, since provisioning depends on a watchdog server
        • Set BUILD param to update to build the module and deploy it to existing watchdog server
    • bootstrap-myapp-server
      • Example to build and provision a customer myapp server.
        • Set CUSTOMER param to customerone to build job with customizing in platform-archetype-bootstrap/customerone/bootstrap-myapp-server
        • Set BUILD param to install to provision a new server using Vultr
        • Set BUILD param to update to update an existing server to a new version
    • bootstrap-myapp-cd
      • Example for continuous delivery job. Updates all myapp servers that run compatible, old version to a new release version

Cheatsheet

Keystores

  • Java keystore list certificates
    keytool -v -list -keystore keystore.pfx
    
  • Java keystore import existing keystore (see below how to create PFX from PEM files)
    keytool -importkeystore -deststorepass 123 -destkeystore keystore.pfx -srckeystore tmpkeystore.pfx -srcstoretype PKCS12
    
  • Java keystore remove certificate
    keytool -delete -alias server -keystore keystore.pfx
    
  • Install OpenSSL, eg. see bootstrap definition platform/bootstrap/bootstrap-common/src/main/resources/bootstrap/definitions/openssl.json
  • Configuration
    set RANDFILE=C:\data\openssl.rnd
    set OPENSSL_CONF=C:\tool\httpd\conf\openssl.cnf
    
  • Convert PEM certificate and key to Java keystore
    openssl pkcs12 -export -in server.crt -inkey server.key -out tmpkeystore.pfx -name server -password pass:123
    
  • Export PEM certificates from Java keystore
    openssl pkcs12 -in keystore.pfx -out keystore.pem -nodes
    
  • Self signed CA and server certificate with SAN
    • Adapt configuration in openssl.cnf
      [ req ]
      req_extensions = v3_req
      [ v3_req ]
      basicConstraints = CA:FALSE
      keyUsage = digitalSignature, keyEncipherment
      subjectAltName = @alt_names
      [ alt_names ]
      DNS.1 = myapp.com
      [ v3_ca ]
      authorityKeyIdentifier = keyid,issuer
      basicConstraints = critical,CA:true,pathlen:3
      keyUsage = critical, cRLSign, keyCertSign
      nsCertType = sslCA, emailCA
      subjectKeyIdentifier = hash
      
    • Generate certificate
      openssl genrsa -out ca.key 2048
      openssl req -new -x509 -extensions v3_ca -days 720 -key ca.key -sha256 -out ca.crt
      openssl genrsa -out server.key 2048
      openssl req -extensions v3_req -sha256 -new -key server.key -out server.csr
      openssl x509 -req -extensions v3_req -days 720 -sha256 -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -extfile C:\tool\httpd\conf\openssl.cnf
      
  • Links

Java

  • Create runtime from JDK
    • Download JDK
    • Generate list of all .jmod files in jmod directory
      • Eg. with ls or java --list-modules
    • Run jlink
      • Start shell in bin directory
      • Replace XXX below with comma separated list of all modules (without .jmod)
      • Linux
        ./jlink --module-path ../jmods --add-modules "XXX"  --output /var/lib/platform/tool/java17
        
      • Windows
        .\jlink.exe --module-path ..\ --add-modules "XXX" --output C:\tool\java17
        
    • Create 7z archive from content of java17 directory
      • Linux java17-nix.7z
      • Windows java17-win.7z
    • Upload to platform dev server
  • Zulu Mission Control
    • Open zmc.ini and add path to Java as first argument
      • Linux
        -vm
        /var/lib/platform/tool/jdk17/bin
        
      • Windows
        -vm
        C:/tool/jdk17/bin/server/jvm.dll
        

Links

Logging

  • Java
    • Tests in platform
      • framework/util/util-core-test/src/main/resources/logback-test.xml
    • Tests in Engine modules
      • myapp-test/src/main/resources/logback-test.xml
    • In general
      • Add a logback.xml (or logback-test.xml) to src/main/resources
    • Runtime
      • For meta-engine in engine.json to overwrite the level of a logger
        "logging": {
            "loggers": {
                "ch.rswk.meta.engine": "DEBUG"
            }
        }
        
  • JavaScript based on util-web
    • In config.js set debug: true
    • QUnit tests
      Logger.useDefaults({
          defaultLevel: Logger.DEBUG
      });
      
    • Clients using meta-client/logging with meta-admin
      • Notifications > Send client notification
        • Any channel the client is subscribed to
        • Action INFO
        • Data {"logLevel": "DEBUG"}

Lucene

  • Inspect index with Luke
    • Download latest Lucene distribution
    • Extract ZIP and set JAVA_HOME in Luke shell scripts
      • Linux luke/luke.sh set $JAVA_HOME/bin/java
      • Windows luke\luke.bat set %JAVA_HOME%\bin\java
    • Run shell script
  • Re-index existing index, eg. after new Lucene version
    • Use clone task from EngineUpdater
      • Using engine-clone.json from a bootstrap module
      • Using IDEA JUnit MainTest.update test method with changed newMetaServiceTestConfig (see below)
  • Re-index existing index, eg. after changes to meta model
    • Implement MetaUpdater/MetaUserUpdater to migrate existing entities with the clone task
    • See EngineUpdaterSchemaTest.java
  • Test with existing index
    • Get copy of index you would like to run locally
      • Manually create a 7z-file from an existing work directory
      • Or backup and download index with meta-admin > File > Search backups
    • Upload in local meta-admin > File
      • Enter correct tags, eg. for MetaService: backup,ch.rswk.meta.engine.MetaService
    • Use restore function on uploaded file
  • Alternative way to test with existing index
    • Extract/copy index to a local directory, eg. C:/tmp/index/work
    • In MetaServiceTest.newMetaServiceTestConfig, set luceneIndexDir to parent directory, eg. C:/tmp/index
    • When running MetaServiceTest, make sure to disable cleanup in teardown

Maven

  • Default goals for platform parent are set to clean install
  • Only build current module and not full reactor mvn -N
  • Run specific test
    • mvn -DfailIfNoTests=false -Dtest=MyTest
    • mvn -DfailIfNoTests=false -Dtest=ch.rswk.meta.core.*.*Test
  • Skip tests mvn -DskipTests
  • Only build certain modules mvn -pl meta-core,meta-client
  • Only build certain module, including dependencies mvn -pl meta-engine -am
  • List module dependency tree mvn dependency:tree
  • Upload third party artifact to Nexus
    mvn deploy:deploy-file -DgroupId='ch.othercompany' -DartifactId=theframework -Dversion='1.0.0' -Dpackaging=jar -Dfile='theframework.jar' \
    -DgeneratePom=true -DrepositoryId=dev-server -Durl=https://dev.mycompany.ch/nexus/repository/thirdparty/
    
    • Use -DpomFile='theframework.pom' instead of -DgeneratePom=true to provide your own POM

Vultr

  • Account
    • API
      • Enable API and store key somewhere safe
        • Used in provision config
      • Add IP/subnets that are going to call the API to access control (dev machines, dev servers, etc.)
    • SSH Keys
      • Generate new keys
        • Linux
          • ssh-keygen -f ~/.ssh/vultr_rsa
          • cat ~/.ssh/vultr_rsa.pub
        • Windows
          • Bitvise SSH Client > Client key manager > Generate new
          • Enter passphrase > Generate
          • Export > Export public key > OpenSSH format
      • Add SSH Key
      • Copy id query parameter from URL and store somewhere safe
        • Used in provision config
  • Products > Firewall > Add Firewall Gruop
    • Accept SSH from your IP
    • Accept MS RDP from your IP
    • Accept HTTP, HTTPS from anywhere
    • Copy Group ID and store somewhere safe
      • Used in provision config
  • Products > Network > Reserved IPs
    • Add for domains with fixed DNS A records and store
    • Copy id query parameter from URL and store somewhere safe
      • Used in provision config

Web

  • Development
    • In WebDriverTest, put breakpoint on first line after runUiTest(...). In IDEA, make sure to set breakpoint Suspend to Thread
    • Use Chromium/Firefox instance opened by the WebDriverTest for client development
    • If you need to access any service
      runtime.getServices(MockGatewayService.class).get(0)
      
  • Service workers
    • Force reload and refresh without using dev tools
      • In config.js/config.json change version
      • In worker-init.js change version in comment on first line
  • URL for tests with ch.rswk.util.web.test.WebServer
    https://localhost.platform.rswk.ch:8088/qunit.html
    
  • URL for tests with meta-engine and HttpService
    https://localhost.platform.rswk.ch:9090/
    
  • Execute action via REST-like API from shell
    • Linux Bash
      #!/bin/bash
      EXECUTE_REQUEST='{"actionId":"myaction","param":{},"reqt":"ecq"}'
      RESPONSE=$(curl -s --fail --header "Accept:application/json" \
      --header "Authorization:Bearer $(curl https://myapp.mycompany.ch/rs/token -s --data "user=myuser" --data "password=pw123")" \
      --header "Content-Type:application/json" --data "$EXECUTE_REQUEST" \
      --request POST https://myapp.mycompany.ch/rs/execute)
      if [ $? -eq 22 ]; then
          echo "Error"
      else
          echo "$RESPONSE"
      fi
      
    • Windows PowerShell/Terminal
      $executeRequest = @{
          actionId = 'myaction'
          param = @{}
          reqt = 'ecq'
      }
      $headers = @{
          'Accept' = 'application/json'
          'Authorization' = $("Bearer " + (Invoke-WebRequest -Body @{user='myuser';password='pw123'} -Method 'POST' -Uri https://myapp.mycompany.ch/rs/token))
          'Content-Type' = 'application/json'
      }
      try {
          $response = Invoke-WebRequest -Body ($executeRequest | ConvertTo-Json) -Headers $headers -Method 'POST' -Uri https://myapp.mycompany.ch/rs/execute
          echo $response
      } catch {
          $err = $_.Exception
          echo $err.Response
      }
      

Software references

Disclaimer

  • Highcharts is only free for non-commercial use. If you plan to use it in a commercial setting, please buy a license
  • SMS delivery in GatewayService depends on an external provider based in Switzerland: iNetWorx SMS-Gateway. Please contact them to order a SMS package. No affiliation with iNetWorx, just happy customers for many years
  • Automated provisioning of bootstrap modules depends on having an account with Vultr. We would like to add more providers in the future, but are happy right now with the simplicity, functionality and pricing provided. Again, no affiliation with Vultr

DevOps

Backend

Frontend