Other

• Update to latest Lucene version
• Meta
  • meta-engine
    • FileService
      • Fix large file handling in FileDownloadResource
      • localDownload now returns the path to the local file, instead of making a copy
• Util
  • util-core
    • SevenZip
      • Use native tar command on Linux and remove tar support on Windows

Other

• Switch to new development server dev.rswk.ch

Other

• Update to latest Logback, OpenSSL versions
• Remove obsolete remaining sync code (mostly Syncthing)
• Remove Wireguard configs
• Util
  • util-web-test
    • AbstractWebDriver
      • Expected name of chrome driver binary changed to chromiumdriver/chromedriver (for compatibility with new dev server setup)

Other

• Bootstrap
  • bootstrap-common
    • openssl.json
      • Update to 3.1.4
  • bootstrap-dev-vm
    • Update IntelliJ configs
• Meta
  • meta-client
    • Fix i18n
  • meta-engine
    • EngineUpdater
      • Add optional argument to CloneTask that specifies how many month of MetaLog to migrate. Eg. -update clone 12 to keep the last 12 month of logs
      • Refactor searching/creating meta entities back to while loop instead of recursion for performance reasons

Other

• Bootstrap
  • bootstrap-common
    • openssl.json
      • Update to 3.1.3
  • bootstrap-dev-vm
    • Update IntelliJ configs. Now allows use of IntelliJ for front- and backend work
• Meta
  • meta-admin
    • Fix missing searcher component
  • meta-engine
    • LuceneService
      • Increase queue size from 1000 to 5000
    • UserService
      • Batch group updates when deleting a user
    • WatchdogService
      • If check is enabled, register FileExpirationNotification listener that removes grants from users for expired files
• Util
  • util-web
    • Retire web-lint.js script
    • Update Bootstrap versions

Other

• Bootstrap
  • bootstrap-common
    • dev.json
      • Update Maven to 3.9.4. Requires change in settings.xml due to HTTP getting blocked with 3.8.1

        // Replace this
        <url>http://central</url>
        // with this
        <url>https://central</url>
        

• Util
  • util-web
    • Remove shell scripts. Just call npm directly from Maven build
    • Use define instead of Maven filtering, eg. for version
    • Use esbuild-plugin-copy instead of Maven to copy resources
  • util-web-test
    • Add NukeServlet that uses Clear-Site-Data header for cleaning up browser cache between test cases
    • runUiTest now takes a Callable and tests must provide their own FluentWait

      final var wait = newFluentWait(driver, DEFAULT_TIMEOUT);
      runUiTest(driver, () -> {
          // Test here
          return null;
      }, url, null);
      

Other

• Bootstrap
  • bootstrap-common
    • openssl.json
      • Update to 3.1.2

Other

• Bugfixes
  • Fix config.js/config.json handling with regards to external in esbuild
  • Fix path to config.json in bootstrap-watchdog-server
• Util
  • util-web
    • web-build.bat/.sh
      • Remove check that a web-build.mjs script exists, now just executes npm run build
      • Remove automatic lint step
    • Rename lint script from web-lint.mjs to web-lint.js

Other

• Switch to SemVer due to NPM modules (was 32.0, is now 2.32.0)
• Web modules are now based on ES6 modules
• Meta
  • meta-client
    • client
      • Now takes single engine argument
    • objects
      • Remove fromMetaObjectVm
      • Rename onAddObject to onAdd
      • Rename onRemoveObject to onRemove
      • Rename toMetaObjectVm to toVm
• Util
  • util-web
    • Add workerfactory.js for service workers
    • Rewrite all web build scripts

Other

• Bugfixes
  • Missing TOTP after clone
  • Missing sorted field for MetaUserConfig Id

Other

• Minor maintenance release
• Meta
  • meta-engine
    • EngineEventBus
      • Use observeOn instead of subscribeOn

Other

• Util
  • util-web
    • nav
      • Params now sorted

Other

• Bootstrap
  • bootstrap-common
    • dev.json
      • Update fossil to 2.21
      • Update Maven to 3.9.2
      • Update Node.js to 20.2.0
• Meta
  • meta-client
    • login
      • Clean up login modals
    • objects
      • Rename clearObjects to clear
  • meta-client
    • Model/model
      • Add parameter additionalClauses to newMultiValueQuery to correctly partition the query without exceeding MAX_CLAUSE_COUNT
  • meta-engine
    • HttpService
      • Add nightly reload of keystore (makes Certbot pre/post scripts obsolete)
      • Add certificate notAfter date health check
      • Add certificate notAfter date to service info
    • Remove all cluster functionality, including SyncService
  • meta-test
    • HttpServiceTest
      • newHttpServiceTestConfig now has two options for setting CSP header
        • cspHeader for a full custom CSP header
        • cspSrc for additional sources using HttpServiceTest.CSP_HEADER_FORMAT
• Util
  • util-web
    • Switch to standard npm command in web-build.sh
    • Update to Bootstrap 5.3.0

Model

• WatchdogStatus
  • Fields are now in ASCII order

Config

• UserServiceEnforceConfig
  • Add otpRequired
  • Rename requireOtp to otpTypes

Other

• Meta
  • meta-admin
    • config.json
      • Add otpRequired
      • Move login related properties to seperate login object
      • Rename requireOtp to otpTypes
  • meta-client
    • login
      • Add webAuthnEnable parameter
      • Redesign recover user and password reset
      • Rename enableEditUser to editUserEnable
      • Remove enableRecoverUserEmail/enableRecoverUserPhone/requireOtp. Now controlled by otpRequired/otpTypes
  • meta-engine
    • LuceneService
      • Properly close old reader on reader refresh
    • UserService
      • resetPassword now accepts 2FA email/phone and email/phone from user config
  • meta-test
    • Fixture
      • Add watchdogConfig to newTestEngineConfig. Use new WatchdogServiceConfig(null, null) to disable watchdog service for testing
• Util
  • util-web
    • crypto
      • Add newUuid that uses randomUUID and falls back to old random string mechanism
      • Remove generateRandomString
      • Rename generateRandomInt to getRandomInt

Protocol

• RecoverUserRequest/RecoverUserReply
  • Renamed from ForgotUserRequest/ForgotUserReply
• FileSearchRequest
  • Add sort, sortReverse options. Use Model.Lucene.FILE_FIELD_UPLOAD_TIMESTAMP and sortReverse:true to get same result as before

Config

• UserServiceConfig
  • Rename forgotContact to recoverContact
  • Rename writeContact to verifyContact

Other

• Update various dependency and plugin versions
• Meta
  • Update to latest Lucene version
  • meta-client
    • Renamed forgotUser to recoverUser
    • login
      • Redesign recover user/password reset
      • Remove enableTokenAuth. Now enabled by default
      • Rename enableForgotUserEmail/enableForgotUserPhone to enableRecoverUserEmail/enableRecoverUserPhone
    • objects
      • Rename searchLoaded to noOfSearchResults
    • objectscache
      • Removed
    • searcher
      • Add classes to allow for styling
  • meta-engine
    • Cleanup dagger modules, see platform-archetype for refrence
    • AbstractEngineWorkQueueService
      • Rename processWorkUnit to offerWorkUnit. Previous offerWorkUnit method was removed
      • Rename process to processWorkUnit, including before and after methods
    • EngineRuntime
      • Add generic FileWatcher interface (AbstractConfigWatcher implements this)
      • Add registerFileWatcher to allow services to watch for file changes
    • EngineUpdater
      • Remove LoggingService from update services and delete logging index after tasks complete
      • Remove fixtures from update services
    • FileService
      • Remove default sort and use new sort fields from FileSearchRequest
    • LuceneService
      • Replace search/write with a single process method
      • Consolidate SearchContext/WriteContext classes into a single LuceneContext
    • MetaLucene
      • Use new DoubleField/IntField/LongField. These fields are automatically sortable, see here
    • MetaSchema
      • Fix missing KeywordAnalyzer for META_LOG_FIELD_DATA_CHANGE, META_OBJECT_FIELD_FIELD
      • Map integer fields to IntField in lucene
      • Use LocalDate for date fields
    • MetaService
      • Remove fixture mechanism. This means MetaAction and MetaType need to be injected in dagger modules. In addition, they are not persisted anymore and can't be created, updated or deleted with write
    • UserService
      • Allow users to login using 2FA email/phone and email/phone from user config. User name takes precedence, then 2FA and lastly user config. Login fails if 2FA or user config search return more than one match
      • Remove fixture mechanism. For testing, insert required config/group/users before running any tests. E.g. using UserService, FixtureImporter.createUserFixtures, etc.
      • startVerifyOtp
        • contact now optional. Uses verifyContact by default
    • WatchdogService
      • checkEngines now uses MetaLog to check if engine is alive
  • meta-test
    • Add EngineTestModule as helper that includes all required engine modules to run server
    • Cleanup dagger modules, see platform-archetype for refrence
    • Fixture
      • Add MetaType fixtures, eg. TEST_META_TYPE. These can be used via MetaServiceTestFixtureModule
    • UserServiceTest
      • Remove getTestUser. Test users, eg. UserServicetest.USER, are now MetaUser
• Util
  • util-core
    • Files
      • Rename resolveIdentifierAbsolutePath to findIdentifierAbsolutePath
  • util-web
    • ui
      • Add theme support (see meta-admin for examples)

Other

• Bootstrap
  • bootstrap-core
    • Cd
      • dryRun now builds the modules to catch any errors
    • Provision
      • Fix parseSchedule
• Util
  • util-web
    • ext
      • simpleSort now uses _.get and supports paths

        searchReply.metas.sort(ext.simpleSort.bind(undefined, ["value", "description"]))
        

Other

• Meta
  • meta-client
    • client
      • Fix download never resetting activeDownloadProgress after first call
  • meta-engine
    • EngineRuntime
      • getServiceInfo now contains more information about the system
    • WatchdogService
      • WatchdogStatus.info now contains all of runtime.getServiceInfo
• Util
  • util-core
    • Env
      • Rename NO_OF_THREADS to THREADS
      • Rename SYSTEM_OS to OS
    • Strings
      • Rename DEFAULT_LOCALE to LOCALE

Other

• Meta
  • meta-engine
    • Invalidate all caches (MetaService, UserService) after backup restore
    • ActionService
      • Clear up what MetaAction schemas do
        • If paramSchema is null, ExecuteRequest.param is ignored and null is passed to action delegate
        • If resultSchema is null, return the action delegate's result. If resultSchema is not null, always validate the result

Other

• Meta
  • meta-engine
    • HttpService
      • Remove hard coded default-src CSP header

Config

• DatabaseServiceDatabaseConfig
  • Renamed to DatabaseServiceDataSourceConfig
• ProvisionServerConfig
  • Add optional awaitTimeout. Time to await watchdog status after provisioning new instance in minutes
  • Add password (and remove serverPassword from ProvisionProviderVultrConfig)
• WatchdogServiceReportConfig
  • engine field special value to report to self is now localhost

Other

• Remove TEST_ prefixes from various fields in test classes and modules
• Replace ImmutableList/ImmutableMap/ImmutableSet with List.of/Map.of/Set.of
• Bootstrap
  • bootstrap-core
    • Cd
      • Add pending update mechanism to allow running Cd multiple times in case of errors, without affecting already updated servers
    • Provision
      • Add ProvisionServerProvider and implement dummy provider for testability
  • bootstrap-common
    • certbot.json
      • Fix Windows hooks
      • Now uses x64 version of Certbot on Windows
    • dev.json
      • Update to Maven 3.8.7
• Meta
  • meta-core
    • Add builders for all model entities, including helpers to clone existing entities

      final var action = MetaAction.create(null).name("My Action").build();
      final var clonedAction = MetaAction.clone(action).name("My Action 2").build();
      

      const clonedAction = model.metaActionClone(existingAction, {name: "My Action 2"});
      

    • Model
      • Parameters are now in ASCII order
      • Add type schema builder

        final var schema = createTypeSchema().properties("{...}").required("deviceId").sortable("updateTimestamp").build();
        

  • meta-client
    • objects
      • Now sorts objects after calls to addObject
      • Remove loadMode, orderBy, orderByOrder parameters
        • Pass existing loadById function to get LOAD_MODE.EAGER
      • Add sort parameter. Takes function that implements Array.sort() to sort the objects array in place

        sort: ext.simpleSort.bind(undefined, "deviceName") // Can handle observable properties
        

      • Add loadByMetaId helper

        loadById: objects.loadByMetaId.bind(undefined, client)
        

    • viewmodel
      • Replace newPlainObservableMapping crutch with newObjectMapping crutch. See meta-admin/viewmodel for examples
  • meta-engine
    • BackupService
      • New interface to break dependency on FileService
    • DatabaseService
      • DatabaseServiceDataSourceConfig can now be injected in addition to the ones configured with DatabaseServiceConfig
    • GatewayService
      • Add sendEmail with emailReplyTo, emailSender arguments to allow different values from config
    • MetaService
      • Fix deleteRecursive
  • meta-test
    • Rename MockGatewayService to GatewayMockService
    • Rename GatewayServiceTestMockModule to GatewayServiceMockModule. Makes gatewayConfig in component builder obsolete
    • Add BackupServiceMockModule. If FileService is not used in any service, include this module instead of FileServiceModule. Makes fileConfig in component builder obsolete
    • Add SyncServiceDummyModule. Makes syncConfig in component builder obsolete
• Util
  • util-core
    • EnvNix/EnvWin
      • Schedule for runTaskInternal is now converted to local date and time
  • util-web
    • require-config.js/web-build.json
      • Instead of including web-build.json from other modules, directly reference the dependency from the modules lib directory

        bootstrap: "../lib/util-web/lib/bootstrap.bundle.min"
        

      • Same for other dependencies like alameda or qunit in index.html/qunit.html

        <script defer src="lib/util-web/lib/alameda.js"></script>
        

    • web-build.bat/.sh
      • Now takes optional third parameter for additional Node modules

        <argument>@node-minify/core @node-minify/uglify-js</argument>
        

      • Custom build scripts get parameter with Node module folder

        const nodeModuleDir = process.argv[2];
        const minify = require(nodeModuleDir + "@node-minify/core");
        

    • Optimize web-dist.xml assembly and JAR files for web modules
    • Replace lodash with underscore
      • get, has now takes array instead of string
      • Merge is now ext.merge(...)
      • Some methods are not availbale in underscore
    • Update Bootstrap, Bootstrap Icons version
    • Update i18next version, see changes to pluralization

      text_with_options_one: "There is a single text for {{name}}.",
      text_with_options_other: "There are {{count}} texts for {{name}}."
      

    • dates
      • New module to break dependency on luxon
      • Add nowDateString (compared to ext)
    • ext
      • Remove
        • now (now dates)
        • setZoneUtc (now dates)
        • parseDateTime (now dates)
        • parseDateTimeInputs (now dates)
        • nowDateTimeString (now dates)
        • isValidDateTime (now dates)
        • toDateString (now dates)
        • toDateTimeString (now dates)
        • setDateTimeInputs (now dates)
        • getDateRange (now dates)
        • insertDateTimes (now dates)
        • extractExifPosition (now img)
        • shrinkImage (now img)
        • orderBy
      • Rename chunkIteration to chunkAsync
        • Remove index parameter
        • func now gets value and index
      • Add
        • merge based on mergician
        • simpleSort
    • i18next
      • init now takes optional options for additional i18next options
    • img
      • New module to break dependency on EXIF
  • util-web-test
    • Add WebDriverTestIT as example for end-to-end test using the final distributed version of a client
    • Update to latest QUnit version

Config

• WatchdogServiceReportConfig/WatchdogServiceCheckConfig
  • Add optional schedule

Other

• Bootstrap
  • bootstrap-common
    • certbot.json
      • Remove directory hooks and configure hooks directly with certonly parameters
• Meta
  • meta-engine
    • EngineUpdater
      • Backup all services before running any tasks
• Util
  • meta-core
    • Wget
      • newHttpClient now defaults to a QueuedThreadPool
      • Rename asString to stringContent

Protocol

• ErrorReply
  • Remove timestamp
• Add ServiceInfoRequest/ServiceInfoReply

Other

• Update various Java dependency, Maven plugin and dev tool versions
• Boostrap
  • bootstrap-core
    • Main
      • Remove buildversion option. Switch to jq utility and Maven build to get matching version of bootstrap-core JAR in Jenkins build
  • bootstrap-dev-server
    • Add jq utility
• Meta
  • meta-admin
    • Now uses service info to populate file component's backup options
  • meta-engine
    • DatabaseService
      • Rename lookupDatabase to jdbi
      • Add backup/restore for SQLite databases
    • EngineRuntime
      • Add getServiceInfo that queries all services implementing InfoProvidingService
    • EngineServices
      • Add InfoProvidingService interface
    • GatewayService
      • WebhookRequest now contains the full HttpRequest to send. Use gatewayService.httpClient().newRequest(...) to create one
    • HttpService
      • Add dispatches to FilterConfig
• Util
  • util-web
    • Update to latest Bootstrap version
  • util-web-test
    • Add remote driver support to AbstractWebDriver

Config

• Bootstrap
  • BuildConfig
    • Add executeWget
  • ExecuteConfigWebhook
    • Remove url
    • Add engine. Watchdog update status webhook now only requires the engine URL, eg. watchdog.mycompany.ch
  • ProvisionProviderVultrConfig
    • Add tags
• Meta
  • LoggingServiceAlertConfig
    • Remove emailSubject, smsMessage

Other

• Update to latest Java 17 version
• Boostrap
  • bootstrap-core
    • Build
      • Remove offline option. Now part of BuildConfig
      • New executeWget option unpacks final archive prior to running WgetExecutor to correctly resolve all definitions
    • Cd
      • Provides continuous delivery functionality using Build and Provision. See Jenkinsfile-myapp-cd in platform-archetype-bootstrap
  • bootstrap-common
    • Move firewall rules in various definitions before service startup
    • Remove userProfile property and use OS specific variables instead (SUDO_USER/username)
    • syncthing-server.json
      • Fix config folder location
• Meta
  • meta-admin
    • Fix file download and handling of new objects in editor
  • meta-client
    • RestClient
      • Rename executeRequest to send
      • Rename executeFileRequest to downloadFile
      • Rename executeRequestAsync to sendAsync
  • meta-engine
    • EngineRuntine
      • Scheduled methods now use Duration
    • WatchdogService
      • Now invokes Protocol.WATCHDOG_UPDATE_RESOURCE webhook when processUpdate fails preparing an update
• Util
  • util-core
    • Wget
      • Add sendAsync helper

Config

• ExecuteConfig
  • Remove owner

Other

• Util
  • util-web-test
    • AbstractWebDriver
      • Make runUiTest window size optional
      • If abstractwebdriver_headless env var is set, tests now run in headless mode

Config

• GatewayServiceConfig
  • Add optional smsGatewayAccount. Can be used for customer-side accounting with the SMS gateway
  • Add smsGauge. If enabled, registers metric gauge for remaining SMS

Other

• Meta
  • meta-engine
    • FileService
      • Fix duplicate check for pending uploads
• Util
  • util-web
    • ext
      • loadConfig now passes all additional arguments to config function

Model

• MetaUserWebAuthnCredential
  • Fix field ordering

Protocol

• ClientNotification
  • Add optional context object, eg. for synced flag set by SyncService
• UserWriteRequest
  • Fix field ordering

Other

• Meta
  • meta-engine
    • SyncService
      • Fix unnecessary ClientNotification processing when running with NoEtcdDelegate
      • Use new context object for marking synced ClientNotification events

Protocol

• LoggingEntry
  • Fields are now in ASCII order

Other

• Remove obsolete @JsonCreator constructors in Java records
• Bootstrap
  • bootstrap-common
    • Remove engine-clone.json. Use engine-main.json directly, see engine-server-update.json
    • engine-server-update.json
      • Remove skipClone
      • Add engineUpdate flag that runs update tasks specified in updateTasks property

        "flags": ["engineUpdate"],
        "properties": {"updateTasks": "clone"}
        

• Meta
  • meta-core
    • model, protocol
      • Remove factory objects
  • meta-client
    • objects
      • Support observable typeId
      • Add onAddObject and onRemoveObject
  • meta-engine
    • EngineUpdater
      • Rename FileServiceTask to FileServiceStorageTask with task ID filestorage
    • FileService
      • Handle duplicate files for pending uploads
• Util
  • util-core
    • Files
      • Rename cleanUp to cleanup
    • SevenZip
      • extract with directory now creates missing parent directories before move to target directory
    • StaticCurlyBracesResolver
      • Rename putAllResolved to putAllAndResolveAll. Now resolves all known variables after adding new ones
  • util-web
    • device
      • Remove factory object
  • util-web-test
    • AbstractWebDriver
      • Remove navigateAndScreenshotComponent
      • Rename runJavaScript to executeScript
      • Add executeScriptAsync, navigateToComponent, setScriptTimeout

Model

• MetaUser
  • MetaUserWebAuthnCredential
    • Add displayName

Protocol

• UserWriteRequest
  • Renamed webAuthnCreate to webAuthnRequest and is now object with displayName
  • Renamed webAuthnCreateReply to webAuthnReply
• UserWriteReply
  • Renamed webAuthnCreateRequest to webAuthnRequest
• SyncReplicateRequest/SyncStatusRequest
  • Replace key with delegateId

Other

• Update to latest Java 17 version
• Meta
  • meta-client
    • component/login
      • Support multiple WebAuthn credentials
  • meta-engine
    • General fixes and improvements for clustered runtimes
    • Remove checkRunning from abstract service classes
    • FileService
      • Remove Set<FileInfo> fixtures. Fixtures were not properly implemented and could only be used for testing. Use FileService.importFileInfos with FileServiceTest.newFileFixtureSupplier for testing

        static void importFixtures(final FileService fileService) throws Exception {
            FileService.importFileInfos(ImmutableList.of(TEST_FILE_SUPPLIER.apply(fileService.config())), fileService);
            final var importer = new FixtureImporter(newTestClient());
            ...
        }
        

    • MetaLucene
      • Add escapeQueryPart
    • UserService
      • Support multiple WebAuthn credentials
      • generateUserToken now rejects users with OTP/WebAuthn, since authentication is only based on password
• Util
  • util-web
    • Update to latest Bootstrap beta version
    • device Add camera permission check in getCameras
    • ext
      • Fix fileEndings/fileFormats checks in loadFileContent
      • Add setZoneUtc

Model

• Model.TYPE_SCHEMA_DEFINITIONS now references $defs field instead of obsolete definitions

Other

• Bootstrap
  • bootstrap-common
    • engine-import.json
      • Add property importFixture for fixture file
• Meta
  • meta-engine
    • GatewayService
      • DeliveryReply now contains the request

Config

• ExecuteConfig
  • Add tmpDir

Other

• Bootstrap
  • bootstrap-core
    • Build
      • In execute script, set java.io.tmpdir to module directory
  • bootstrap-common
    • Set java.io.tmpdir to {tmpDir}/{tmpDirWin} in all definitions that call java
    • certbot.json
      • Call pl-deploy.ps1 after initial certbot run on Windows
• Util
  • util-core
    • Switch from jackson-module-afterburner to jackson-module-blackbird
    • Env
      • Revert runTask schedule date format on Windows

Protocol

• SearchRequest/SearchReply
  • Fields are now in ASCII order
• WriteObjectUpdateRequest
  • Fields are now in ASCII order
• WriteRequest
  • Fields are now in ASCII order

Other

• Bootstrap
  • bootstrap-core
    • CmdExecutor
      • Fix exception handling and reduce DEFAULT_TIMEOUT to 5 minutes
  • bootstrap-common
    • certbot.json
      • Fix certbot.exe call on Windows
• Meta
  • meta-engine
    • MetaService
      • Catch JsonSchemaException during start-up, so service doesn't fall over, if there are actions/types with old schemas
    • WatchdogService
      • Handle errors in processUpdate
• Util
  • util-core
    • Env
      • runTask now returns EnvExecuteResult
      • Fix runTask schedule date format on Windows
    • SevenZip
      • Parameters are now in ASCII order
  • util-web
    • web-build.json
      • Remove top level lib
      • Replace array with proper object to specify custom file names

        {
          "include": [],
          "libs": [
              {
                  "name": "exif.js",
                  "url": "https://cdn.jsdelivr.net/npm/exif-js"
              }
          ]
        }
        

Config

• BuildConfig
  • Remove bootstrap, propertiesPath
  • Add bootstrapConfig

Other

• Bootstrap
  • Try to reduce duplication in build and bootstrap configs
  • bootstrap-core
    • Build
      • Change run arguments
        • Remove mvnsettings option. Now always uses PLATFORM_DATA_DIR/settings.xml
        • Rename mvnrepodir to mvnrepo
        • Add merge option. If provided, merges the given JSON file with bootstrapConfig from BuildConfig. Merge operation must result in a valid ExecuteConfig
• Meta
  • meta-engine
    • GatewayService
      • Fix SMS message encoding error
• Util
  • util-web
    • i18next
      • Handle keyval errors to make sure module works, even if keyval is not available

Config

• HttpServiceHeaderConfig
  • Remove xframe option. Use CSP frame-ancestors if needed. Defaults to:

    frame-ancestors 'none'
    

  • Add patterns, list of HttpServiceHeaderPatternConfig for adding custom headers

Other

• Util
  • util-core
    • Ext
      • newFixedThreadPool now takes minThreads instead of threadsFactor
    • Wget
      • Add stopHttpClient helper method. Tries to stop/destroy client and its executor
  • util-web-test
    • Renew localhost.platform.rswk.ch certificate
• Bootstrap
  • bootstrap-core
    • Build
      • Remove deploy option. Now only depends on deploy in BuildConfig

Model

• MetaUser
  • Fields are now really in ASCII order

Config

• HttpServiceSslConfig
  • cipherSuites, protocols are now optional and use defaults from HttpService

Other

• Util
  • util-web
    • Update Node.js to 17.5.0 for native fetch and adapt web build script
    • Remove copy option from web-build.json
    • web-dist.js is now expected to be called web-dist.json
• Meta
  • meta-client
    • component/login
      • Adapt to new OTP changes in UserService
  • meta-engine
    • EngineUpdater
      • Add FileServiceTask (task ID filemigrate) to migrate to new storage directory layout
    • FileService
      • Files are now partitioned into subdirectories by the first two characters of the key. Even with a lot of files, this should make the directories browsable with a normal file manager

        file
            store
                c2
                    c212645f-9416-42c7-9264-5f9416b2c7b9
        

    • UserService
      • processUserFixtures
        • Now uses hashed true. This makes TOTP work for test fixtures, but also requires passwords to be hashed (use generateScryptHash)
      • getCachedUser/getCachedGroup/getCachedConfig
        • Now return Optional values, so we can cache absence of a value
      • userSearch/userWrite
        • When masked, email/phone are now redacted
      • userWrite
        • OTP validation flow now stores and compares the hash of the user in the requests
        • OTP values can now be changed, eg. to a different email or generate a new TOTP key
• Bootstrap
  • bootstrap-common
    • dev.json
      • Manually download Maven and Node binaries on Linux
  • bootstrap-dev-server
    • Fix various errors

Config

• ProvisionWatchdogConfig
  • Add optional engineUser. User will be granted FILE_DOWNLOAD permission on uploaded bootstrap archives. This makes it possible to have watchdog reporting users with limited permissions

Other

• Meta
  • meta-core
    • Model
      • Add MAX_CLAUSE_COUNT
      • Add newMultiValueQuery (was previously in MetaLucene)
  • meta-client
    • component/login
      • Replace enableForgotUser with enableForgotUserEmail, enableForgotUserPhone
  • meta-engine
    • EngineRuntime
      • Add scheduler that can be used for caches that rely on expiration. Uses Scheduler.systemScheduler() internally for now

        Caffeine.newBuilder().scheduler(runtime.scheduler()).expireAfterAccess(Duration.ofMinutes(5)).build();
        

      • Add schedule method with Callable
    • UserService
      • forgotUser now sends the names of all existing users and not just the first one it finds
• Bootstrap
  • bootstrap-common
    • certbot.json
      • Add custom renew task on Windows that runs once a day at 01:00

Protocol

• Add AuthTokenRequest/AuthTokenReply
• Add ForgotUserRequest/ForgotUserReply
• LoggingEntrySearchRequest/LoggingEntrySearchReply
  • Fields are now in ASCII order
• LoggingMetricSearchRequest
  • Fields are now in ASCII order
  • query is now optional
• LoggingMetricSearchReply
  • Fields are now in ASCII order
• LoginRequest
  • Add optional token for token based login
• LoginReply
  • Add client, user, version fields that contain sanitized values from request

Config

• UserServiceConfig
  • Remove keySize from UserServiceTokenConfig
  • Add forgotContact

Other

• Util
  • util-core
    • Strings
      • Add redactEmail, redactPhone
  • util-web
    • ext
      • Add redactEmail, redactPhone
• Meta
  • meta-client
    • Add forgotUser
    • client
      • Remove engine observable
      • Check for correct request type in all request/reply methods
    • component/login
      • Remove enableSaveLogin option and UI elements
      • Add enableForgotUser, enableResetPassword options
      • Add enableTokenAuth. If enabled, after successful authentication the client will get a auth token and store it in local storage. Token is restored when login component is loaded and user is logged in (if token is still valid)
      • Add WebAuthn error toast
  • meta-engine
    • UserService
      • recordFailedAttempt
        • Remove cause
        • Add type. Must be one of AttemptType
      • Rename VerifyTokenResponse to VerifyTokenReply
      • Index OTP email, phone, webhook in MetaUser Lucene document
      • Index top level textual fields in MetaUserConfig Lucene document
      • Add authToken. Generate auth token that is valid for a specific IP and expires in 12 hours. Can be used in LoginRequest to bypass normal login/auth flow
      • Add forgotUser. Can recover user name from email/phone in OTP settings or email/phone property from user config
      • Add OTP as available parameter to OTP email subject formatter

Protocol

• Remove BINARY_BUFFER_SIZE constant

Other

• Meta
  • Fix problems with binary WebSocket messages due to buffer/max message sizes
  • meta-core
    • AbstractWebSocketListener
      • sendBytes now takes a ByteBuffer payload so we dont have to wrap/unwrap byte arrays
  • meta-client
    • logging
      • Add installDebugLogger. This logger is intended for devices where you can't open dev tools. It creates a visible HTML element in the current document

        logging.installDebugLogger();
        

  • meta-engine
    • UserService
      • Rename hasUserPermissionToAnyOfKeys to hasUserPermission and revert to using cached groups again for performance reasons

Config

• UserServiceConfig
  • Add authContact, resetContact, writeContact in the form of the new UserServiceContactConfig. These templates are used for the different OTP challenge types and replace otpEmailContent, otpEmailSubject, otpSmsMessage

Other

• Util
  • util-core
    • SevenZip
      • extract now moves directory instead of making a copy and cleans up the created temp directory
• Meta
  • meta-engine
    • Switch all services using Guava's Cache to recommended replacement Caffeine
    • MetaService
      • deleteSearchResultsRecursive now takes optional channel
    • UserService
      • Switch to AsyncLoadingCache. getCachedX methods, hasUserPermissionToAnyOfKeys, isMember, generateUserToken now return ListenableFuture
      • Differentiate between the different OTP challenge types (auth, reset password, verify and write user). OTP challenge is now created for the combination of type, user and IP address. If a open challenge exists, a new one is not generated until the existing one has expired
      • userWrite add ipAddress in addition to existing clientId for OTP verification

Model

• MetaUserConfig
  • Add getConfig helper method

Other

• Meta
  • meta-engine
    • MetaSchema
      • Add support for oneOf/anyOf in getFieldTypeRecursive
    • UserService
      • userWrite
        • Replace verifyOtp with optional clientId. If provided, OTP changes must be verified by a second request with a valid OTP
      • Fix splitting up local/synced write operations
      • Escape name/id queries in case they contain special characters that break Lucene search

Model

• Meta
  • id is now a regular UUID with dashes
• MetaUser/MetaUserGroup
  • Names can now contain all non whitespace characters
• model
  • newTypeSchema takes a parameter object

Protocol

• TokenRequest/TokenReply
  • Removed
  • Tokens are now only available through the /rs/token endpoint

Config

• FileServiceConfig
  • Replace lucene with luceneIndexDir
• HttpServiceConfig
  • Rename resourceBase to resources. Is now a set of directories that are served by the default servlet
• LoggingServiceConfig
  • Replace lucene with luceneIndexDir
  • LoggingServiceRetentionPathConfig
    • Rename filePattern to pattern
• MetaServiceConfig
  • Replace lucene with luceneIndexDir
• UserServiceConfig
  • Replace lucene with luceneIndexDir

Other

• Util
  • util-web
    • Remove configloader module. Use ext.loadConfig(...) instead
    • crypto
      • Refactor with parameter objects
      • Rename randomValues to getRandomValues
    • nav
      • Remove setPersistState. Use nav.init({persistState: true}) instead
      • init and register now take parameter object
  • util-web-test
    • AbstractWebDriver
      • Rename closeVisibleModal to clickVisibleCloseButtons
• Meta
  • meta-client
    • client
      • Fix error with buffer size used for uploads that terminated the WebSocket
    • RestClient
      • Add resetToken
  • meta-engine
    • EngineRuntime
      • Add scheduleDaily. Runs a command at the given hour/minute every 24h
      • Add scheduleNightly. Runs a command every 24h, sometime between 00:01 and 02:59
    • FileDownloadResource
      • Add new {key}/{fileName} endpoint to download content with a given file name. This version accepts the token as a query parameter in Protocol.TOKEN_PARAM (in addition to the header field)
    • FileService
      • Use KeywordAnalyzer for all Lucene document fields
    • LoggingService
      • Use PerFieldAnalyzerWrapper with KeywordAnalyzer as default and StandardAnalyzer for LOGGING_FIELD_MESSAGE Lucene document fields
    • LuceneService
      • LuceneServiceConfig
        • Add analyzerSupplier. Services that use a LuceneService now have to pass in the used analyzers
      • Add reopenIndex. Must be called in case the used analyzers are dynamic and have changed
    • MetaSchema
      • FieldGenerator
        • Add putAnalyzers to set custom Lucene analyzers per field
    • MetaService
      • Use PerFieldAnalyzerWrapper with StandardAnalyzer as default, KeywordAnalyzer for all meta ID Lucene document fields and use additional analyzers from indexing types in MetaSchema
      • deleteSearchResultsRecursive now has optional parameter counter
    • UserService
      • Use KeywordAnalyzer for all Lucene document fields
      • Rename token to generateUserToken with UserTokenRequest
      • Rename GenerateTokenRequest to TokenRequest
        • Rename validSeconds to expirationSeconds and make it optional. Uses UserServiceTokenConfig.expiration when 0
      • Remove getClientEntry. Use getClientList with new helpers on ClientListReply
• Bootstrap
  • bootstrap-core
    • Provision
      • Use awaitWatchdogUpdateStatus instead of awaitInstanceRunning when waiting for new server
  • bootstrap-test
    • AbstractBootstrapTest
      • prepareEngineConfig now takes path to engine config. Use moduleDir.resolve(DEFAULT_ENGINE_CONFIG_PATH) to get the same result as before

Protocol

• AuthReply
  • Add config, groups like we have in LoginReply. If user requires auth, configs and groups are now sent in AuthReply instead of LoginReply
• ClientListEntry
  • Fields are now in ASCII order
  • Rename clientName to client
  • Rename clientVersion to version
  • Add ipAddress
• ClientNotification
  • Remove clientName, clientVersion. These were only used by UserService to populate the client list entries
• ErrorReply
  • Fields are now in ASCII order
• ErrorReplyException
  • Fields are now in ASCII order
  • Switch to record style getters instead of public fields
• ErrorReplyFactory
  • Fields are now in ASCII order
• LoginRequest
  • Fields are now in ASCII order
  • Rename clientName to client
  • Rename clientVersion to version
  • Add optional locale to indicate client preference (eg. for OTP messages)
• Protocol
  • Remove USER_AUTH_ERROR_CHANNEL
• ResetPasswordRequest
  • Add locale
• UserWriteRequest
  • Add optional locale. This is not stored, just used for i18n during OTP verification

Config

• GatewayServiceConfig
  • Fields are now in ASCII order
• LoggingServiceConfig
  • Fields are now in ASCII order
• SyncServiceConfig
  • Fields are now in ASCII order
• UserServiceConfig
  • otpEmailContent, otpEmailSubject and otpSmsMessage now support i18n. otpEmailContent and otpSmsMessage are formatted with one variable containing the OTP

    "otpSmsMessage": {
      "de": "Ihr OTP: %s",
      "en": "Your OTP: %s"
    }
    

• WatchdogServiceConfig
  • Fields are now in ASCII order

Other

• Util
  • util-core
    • Files
      • Add getHashAndSize that returns SHA-256 hash of file and its size
    • Strings
      • resolveI18Templates now checks for the presence of an entry for DEFAULT_LOCALE
      • Add getI18nString to get value from template map that falls back to DEFAULT_LOCALE
• Meta
  • meta-admin
    • Remove export/import on user component
  • meta-client
    • postClientNotification does not set timestamp of passed client notification anymore
  • meta-engine
    • AbstractEngineWorkQueueService
      • Rename tryAddWorkUnit to processWorkUnit
      • Add offerWorkUnit. This variation does not return the processing future
      • Rename onBeforeUnit/onAfterUnit to beforeProcess/afterProcess and add unit parameter
      • Add ShutdownType option
    • EngineRuntime
      • Remove PreparedShutdownService
    • UserService
      • otpEmailContent and otpSmsMessage formats only get one variable containing the OTP
      • Remove posting of client notifications on auth errors
• Bootstrap
  • bootstrap-core
    • Fix errors in certbot definition for Windows

Protocol

• NotificationCacheReply/NotificationCacheRequest
  • Removed

Config

• UserServiceConfig
  • Remove totpValidity. Accept one future and two past TOTPs by default
  • Rename otpEmailMessageFormat to otpEmailContent. Can now contain a file URI that is resolved via Strings.resolveTemplate
  • Rename otpSmsMessageFormat to otpSmsMessage

Other

• Meta
  • meta-engine
    • AbstractEngineExecutionThreadService/AbstractEngineIdleService
      • Now implement new interface ConfigurableService
      • Add startUp and shutDown implementations to register with new event bus for ConfigWatcherNotification. All services should call super.startUp/shutDown
    • EngineRuntime
      • Switch from Guava's EventBus to a custom RxJava based bus in EngineEventBus

        final var sub = runtime.eventBus().subscribe(ClientNotification.class, this::onClientNotification);
        EngineEventBus.dispose(sub);
        

      • Add AbstractConfigWatcher and implement check for config changes (was previously part of NotificationService)
    • EngineServices
      • Add ConfigurableService interface
      • Add ConfigWatcherNotificationConsumer
    • NotificationService
      • Removed. With the removal of notification cache and moving of event bus/config watcher to runtime, this service is obsolete
    • UserService
      • verifyTotp
        • Replace validPeriods with acceptedCodes
      • Support HTML content for OTP emails
      • Delete configs of deleted users and groups
  • meta-test
    • Add ClientNotificationTestSubscription and MetaNotificationTestSubscription

Config

• FileServiceConfig
  • Add expiration in seconds for expiration check schedule

Other

• Util
  • util-web
    • ext
      • Refactor displayNotification, insertDateTimes, loadFileContent, requestPermission, shrinkImage to take parameter objects
      • Add retainAtoZAndDigits
• Meta
  • meta-core
    • Model
      • Add DATE_NUMBER_MAX, DATETIME_NUMBER_MAX constants for range queries
  • meta-engine
    • EngineRuntime
      • Rename shutDownHandle to cancelFuture. Is now static and takes any type of Future
    • EngineUpdater
      • JsonExportTaskConfig/JsonExportTaskResult/EngineImportTaskConfig
        • Fields are now in ASCII order
      • JsonImportTaskConfig
        • Rename various fields
        • Add userUpdate
    • FileService
      • Properly implement expiration mechanism
        • Index expiration timestamp
        • Post FileExpirationNotification when a file has expired and was deleted
    • MetaLucene
      • Add otpType field to user document
    • UserService
      • Fix problem when writing users with existing TOTP keys when hashed is false, eg. in CloneTask
      • Add getClientEntry

Protocol

• PasswordResetReply
  • Add success in case password was reset

Other

• Java
  • Remove final from instanceof pattern matching variables
• Util
  • util-core
    • Ext
      • Add newPoolThreadFactory, newFixedThreadPool, newScheduledThreadPool
    • Wget
      • Remove static HTTP_CLIENT instance that was used for toFile, asString, invokeWebhook. These methods once again start and stop a client for each invocation
      • Remove unsafe option. Pass newUnsafeHttpClientTransport to newHttpClient
      • newHttpClient now sets a default executor
• Meta
  • meta-client
    • component/login
      • Add enableWebAuthn option
      • Smaller improvements and fixes
• Bootstrap
  • bootstrap-dev-vm
    • Fix problem with trusting certificates on Windows

Protocol

• UserWriteReply
  • Add verifyOtp to signal that OTP changes need verification
• UserWriteRequest
  • Add otp when verifying OTP changes

Other

• Merge Nexus repositories bootstrap-snapshots and bootstrap-releases (again) back into a single bootstrap repo, or it gets annoying when switching bootstrap modules between release and snapshot versions
• Util
  • util-core
    • Wget
      • Remove timeout parameters from toFile
      • Remove executeWithHttpClient
      • Add invokeWebhook
  • util-web
    • Update to latest Bootstrap Icons 1.7.0
• Meta
  • meta-client
    • component/login
      • Support new userWrite OTP verification flow. Make sure to include new component/verify component in ui.bind and web-dist.js configs
      • Remove auto login after reconnect behaviour
      • Remove onTotpReply parameter
    • component/verify
      • Reusable verification modal
  • meta-engine
    • UserService
      • userWrite has a new parameter verifyOtp. If true, changes to OTP settings requires a second request with a valid OTP/TOTP for verification
  • meta-test
    • WebDriverTest
      • Add onLoginEmailAuth and onLoginTotpAuth helper methods that can be used together with existing login helper
• Bootstrap
  • bootstrap-core
    • WgetExecutor
      • Remove timeout, unsafe options

Model

• MetaGrantKeys
  • Rename POST_CLIENT_NOTIFICATIONS to POST_CLIENT_NOTIFICATION
  • In Java enum, value is now key
• MetaLog
  • Fields are now in ASCII order

Protocol

• ClientNotification
  • Fields are now in ASCII order
• FileInfo
  • Fields are now in ASCII order
  • Add optional expiration
• PingReply/PingRequest
  • New method for client side pings
• ResetPasswordRequest
  • user not nullable
• SubscriptionRequest
  • Fields are now in ASCII order
  • channels not nullable
• UploadRequest
  • Fields are now in ASCII order
  • Add optional expiration

Config

• FileServiceConfig
  • Replace syncthingApi/syncthingApiKey with syncthingConfig. FileService now retreives the API and key directly from the config file at runtime
• LoggingServiceConfig
  • Remove logPath
  • Add retentionPaths

    "retentionPaths": [
        {
            "filePattern": "{logDir}/*.zip",
            "retention": 14
        }
    ]
    

Other

• Meta
  • meta-admin
    • Fix clients state update handling
    • Fix clients and notification cleanup
  • meta-client
    • Client/client
      • Remove reconnectInterval option and always try to reconnect the WebSocket (except when the client gets kicked due to policy violations)
      • Add ping mechanism, so we have server and client side pings for better detection of lost connections
    • Client
      • disconnect now returns a future to make it behave like the JavaScript client
    • client
      • workerPath now defaults to lib/meta-client/js/, so most apps don't need to provide it anymore
    • RestClient
      • executeFileRequest
        • Remove timeout parameters, since they were only applied to receiving header information in the first place
  • meta-engine
    • EngineUpdater
      • JsonImportTaskConfig
        • Add usersHashed to be able to import users with hashed passwords
    • UserService
      • recordFailedAttempt
        • Add cause
      • userWrite
        • When deleting a user, remove it from all groups
        • Add enforce parameter. If set to false, any UserServiceEnforceConfig is not enforced
    • EngineWebSocketController
      • Allow clients to receive notifications on channel equal to their clientId
      • Only allow password reset for users with login permission
• Util
  • util-core
    • Dates
      • Remove toZonedDateTime, use ZonedDateTime.now()/atZoneSameInstant
    • Files
      • Method parameters are now in ASCII order
      • copyDir now takes a optional VariableResolver. If provided, all files are copied using copyTextFileFiltered with the given resolver
  • util-web
    • Change naming convention for require-init.js script to index-init.js
    • configloader
      • No longer a requirejs plugin
      • Client configs should now be based on a config.json file that is fetched in config.js module. The config module can then be passed to load

        // config.js
        return function fetchConfig() {
            return fetch("./js/config.json").then(
                (response) => response.json()
            ).then(
                (config) => Object.assign({version: "${project.version}"}, config)
            );
        };
        

        // index.js
        configloader.load(config).then(function (loadedConfig) {
            that.config = loadedConfig;
            return ui.bind({
                components: [
                    {module: "myapp-web/component/mycomponent", name: "mycomponent"}
                ],
                i18NextOptions: {
                    debug: that.config.debug,
                    resources: i18n
                },
                viewmodel: that
            });
        }).then(function () {
            that.hasLoaded(true);
            LOG.info(`loaded, version=${that.config.version}`);
        });
        

    • ext
      • Add nowDateTimeString
        • Should replace new Date().toISOString() when a UTC ISO date time string is needed and is equivalent to ext.toDateTimeString(ext.now())
      • registerServiceWorker
        • Remove version and scope. The worker-init.js script is now expected to be in the root directory, so the default scope is sufficient
    • i18n
      • Remove newExtendedI18n. Use _.defaultsDeep to merge multiple i18n resources before passing them to init
    • ui
      • Rename init to bind. Now takes an parameter object
      • Add modalOptions, offcanvasOptions, tooltipOptions to corresponding KO bindings instead of exposing a couple of specific options
      • Add progress KO binding
• Bootstrap
  • bootstrap-common
    • syncthing.json/syncthing-server.json
      • Add required actions to install and configure Syncthing
  • bootstrap-core
    • CopyExecutor
      • filter option now works for directories and applies to all contained files
  • bootstrap-test
    • Finish cluster test case with etcd, Syncthing and Wireguard setup

Model

• MetaGrantKeys
  • Rename ACTION_EXECUTE to EXECUTE and value ac to ex
  • Change POST_CLIENT_NOTIFICATIONS value cln to cn
  • Add SYNC grant for SyncService based operations, that only need to be available to sync peers
• MetaLogType
  • Moved to its own class instead of being inner class of MetaLog
• MetaUser
  • Fields are now in ASCII order
  • Move WebAuthn options from otp to own field webAuthn
  • Rename MetaUserWebAuth/MetaUserWebAuthCredential to MetaUserWebAuthn/MetaUserWebAuthnCredential
• MetaUserConfig
  • Fields are now in ASCII order
  • Remove id and use computed identifier, see Model.forgeConfigId
• MetaUserGroup
  • Fields are now in ASCII order

Protocol

• AuthRequest
  • Rename AuthRequestWebAuthAssertionReply to AuthRequestWebAuthnAssertionReply
• ExecuteRequest / ExecuteReply
  • Renamed from ActionRequest / ActionReply
• FileSearchRequest/FileSearchReply
  • Rename after/last to cursor
• LoggingEntrySearchRequest/LoggingEntrySearchReply
  • Rename after/last to cursor
• LoggingMetricSearchRequest/LoggingMetricSearchReply
  • Rename after/last to cursor
• LoggingMetricTimeRequest/LoggingMetricTimeReply
  • New method for starting/stopping client side metric timers
• LoginReply
  • Remove otp
  • Add isAuth, which tells a client if further authentication is needed. If false, no call to auth with the OTP/2FA is necessary
  • Rename LoginReplyWebAuthAssertionRequest to LoginReplyWebAuthnAssertionRequest
• LogMessage
  • Remove timer/timerRef. Use new LoggingMetricTimeRequest
• PasswordResetRequest/PasswordResetReply
  • Support automatic password reset via verification of user OTP
• SearchRequest/SearchReply
  • Rename after/last to cursor
• UploadReply
  • Add duplicate flag to signal that file with same hash already exists and upload has no effect
• UserConfigSearchReply/UserConfigSearchRequest
  • Add cursor and limit like other search methods
• UserGroupSearchReply/UserGroupSearchRequest
  • Add cursor and limit like other search methods
• UserSearchReply/UserSearchRequest
  • Add cursor and limit like other search methods
• UserWriteReply
  • Rename UserWriteReplyWebAuthCreateRequest to UserWriteReplyWebAuthnCreateRequest
• UserWriteRequest
  • Add webAuthnCreate flag to start registration of new WebAuthn credentials
  • Rename UserWriteRequestWebAuthCreateReply to UserWriteRequestWebAuthnCreateReply
• UserWritePartialReply/UserWritePartialRequest
  • New method for partial update of MetaUser
• UserGroupWritePartialReply/UserGroupWritePartialRequest
  • New method for partial update of MetaUserGroup
• WriteObjectPartialReply/WriteObjectPartialRequest
  • New method for partial update of MetaObject, replaces WRITE_ACTION.PARTIAL_UPDATE option
• WRITE_ACTION
  • Remove PARTIAL_UPDATE
  • See new methods on MetaService and UserService
• Add ResetPasswordRequest/ResetPasswordReply

Config

• BuildValidateConfig
  • Add validatorClass that can be used for custom validation logic of the bootstrap module before build starts. Validator must extend AbstractBuildValidator
• FileServiceConfig
  • Add syncthingApi/syncthingApiKey for clustered file synchronisation using Syncthing
• LuceneServiceConfig
  • Fields are now in ASCII order
• HttpServiceConfig
  • Fields are now in ASCII order
  • Add forceHttps option to be able to force https on certain paths
  • Add redirect option to add RedirectRegexRule
  • HttpServiceHeaderConfig
    • Add xframe option to use different X-Frame-Options header. Default is still DENY
• UserServiceConfig
  • Fields are now in ASCII order
  • Add enforce to set password and OTP rules
  • Add ipAddressAttempts to enable blocking of failed attempts based on IP
  • Add otpEmailMessageFormat, otpSmsMessageFormat
  • Rename webAuth to webAuthn
• WatchdogServiceCheckConfig/WatchdogServiceReportConfig
  • Remove schedule, unsafe options

Other

• Switch from Mercurial/SCM-Manager to Fossil SCM
• Java
  • Update to JDK 17
  • Switch to IDEA as default IDE
    • Source code navigation
    • Remove platform-ide repository
  • Start preparing modules for JPMS
    • Currently unable to switch due to split packages in Lucene, which should be resolved with version 9
    • All split packages were removed, resulting in breaking changes in test modules
• Use Jenkins pipeline for job configurations and add to SCM
• JavaScript code adapted to new JSlint rules regarding parameter and property ordering
  • Instead of using spec parameter object, use object destructuring assignment

    function ({
        param1 = 123,
        param2
    }) { ... };
    

    factory.myObject = function ({prop1, prop2}) {
        return Object.freeze(
            {prop1, prop2}
        );
    };
    

• Util
  • util-core
    • Dates
      • Add today returning the current LocalDate
    • Ext
      • Remove getSha256Hash, use Hashing directly

        Hashing.sha256().hashBytes(...);
        

    • Json
      • Add toJson with LocalDate
    • SevenZip
      • Renamed from P7zip
      • Include 7z executable for Linux
      • Due to 7z not preserving file permissions, run chmod +x for all extracted executables when extracting from a tar archive
  • util-web
    • Build
      • Rework web-build.js to use Node.js promise functions, eg. fsPromises
      • Add automatic JSLint for all files in src/main/resources/js, including any subdirectories
      • Switch to a fork of r.js that uses latest version of Esprima
    • Update to Bootstrap 5
    • device
      • Make entities from event and record factories immutable using Object.freeze
    • i18next
      • init now takes a spec object

        i18next.init({debug: config.debug, resources: i18n})
        

    • nav
      • State is now also considered if a registered update delegate should be invoked
    • ui
      • Add offcanvas KO binding for Bootstrap Offcanvas
      • Add tooltip KO binding for Bootstrap Tooltips
  • util-web-test
    • AbstractWebDriver now uses @ValueSource for parameterized tests, so we can pick which drivers to use
• Meta
  • meta-admin
    • Add metric filter presets and config option
    • Add partial update support for user and groups
    • Use objects/searcher in files and users and implement cursor/limit support
  • meta-client
    • Add resetPassword
    • client
      • Add navigator online status to connection status check
    • objectscache
      • Does not inherit from objects anymore. Pass the objects instance as a parameter
    • component/login
      • Remove auto connect client on load
      • Remove render logic and margins
      • Improve edit user options
      • Add locale switcher
      • Add password reset flow
      • Add customClasses option
      • Add passwordRegex, requireOtp options to enforce password and 2FA rules
  • meta-core
    • Make JavaScript entities in model and protocol factories immutable using Object.freeze
  • meta-engine
    • AbstractEngineLauncher
      • Add newUpdaterServices, so applications can construct UpdaterServices instance with specific modules
    • ActionService
      • Result of execution is not validated against result schema anymore
      • Add ipAddress to delegate arguments
    • ActionExecuteResource
      • Add REST-like API for executing actions
    • DatabaseService
      • Remove getDefaultDatabase method
      • Add checkTableExists helper method
    • FileService
      • Implement sync support and integrate with SyncService
      • Add UPLOAD_FAIL_NOTIFICATION_ACTION, in addition to existing UPLOAD_DONE_NOTIFICATION_ACTION
    • GatewayService
      • Add alert for smsRemaining gauge, hardcoded to 100
    • LogginService
      • Add loggingMetricTime method
    • MetaLucene
      • Grants on users and groups are now indexed
      • Add users field to groups with number of users
    • MetaService
      • Add writeObjectUpdate as replacement for previous WriteAction.PARTIAL_UPDATE
    • UserService
      • Attempts are now recorded and blocked for clientId, user and ipAddress
      • Rename verifyUserOtp to verifyOtp
        • Only verifies the OTP and does not generate a token anymore. Use the now public generateToken method
      • OTP message formats can now make use a second parameter with the expiration timestamp
      • Add groupWritePartial and userWritePartial
      • Add resetPassword
• Bootstrap
  • bootstrap-core
    • Shaded JAR is now a seperate artifact with classifier shaded
    • Build
      • Remove apt install for p7zip-full, so execute can run offline. Still try install at, but ignore failure
    • Provision
      • Update to Ubuntu 21.04 / Windows 2019 as default OS
    • VariableExecutorAction
      • Add overwrite option
  • bootstrap-common
    • certbot.json
      • Switch to beta version of EFF certbot for Windows
    • dev.json
      • Switch to NodeSource repository on Linux for nodejs package
      • Fix problems with Linux installation
    • Add new definitions
      • etcd-server.json
      • openssl.json
      • sqlite.json
      • syncthing.json
      • syncthing-server.json

Config

• Meta
  • SyncServiceConfig
    • peers is now a list of SyncServicePeerConfig
  • SyncServicePeerConfig
    • Add id for the engine ID
  • SyncServiceEtcdConfig
    • Remove authorization and TLS options
• Bootstrap
  • Try to reduce duplication in build and provision configs
  • BuildConfig
    • Add propertiesPath option to reference JSON file with common properties share by multiple build configs
  • ProvisionConfig
    • Move id form ProvisionWatchdogConfig to ProvisionConfig as serverId
  • ExecuteConfig
    • Make dataDir, logDir and toolDir optional. They default to the platform default directories

Other

• Fix problems with Java 16 update
  • Update to 2021-06 Eclipse builds
  • Update to Hirsute Hippo Ubuntu builds
• Start preparing for Java modules
  • Remove split packages
• Start switching to @Nullable from Checker Framework
• Util
  • util-core
    • Dates
      • Remove fromSqlTimestamp
    • P7zip
      • Add support for tar.gz on Windows. Now uses same command for Linux and Windows
  • util-core-test
    • Move all test classes to test package
  • util-web
    • nav has new option to persist state. Use nav.setPersistState(true) before calling init
• Meta
  • meta-client
    • Rename TokenHelper/tokenhelper to RestClient/restclient
• Bootstrap
  • Fix errors where moduleDirWin was not used for Windows specific commands
  • bootstrap-core
    • Build
      • Remove debug option
    • Provision
      • Figures out if it's a new server or an update by checking for a running instance, if a server provider is configured

Model

• WatchdogStatus
  • alerts is a list of WatchdogStatusAlert instead of a Map<String, String>. Using a map caused problems with partial updates of the status objects

Config

• EngineWebSocketControllerConfig
  • Rename allowUnauthorizedLog to logAllowUnauthorized
  • Add uploadGrantFull. If true, grants FILE_FULL on upload instead of FILE_DOWNLOAD

Other

• Update to Java 16
• Discover FluentFuture
• Rename all inner classes that were prefixed with the parent class name
  • Eg. AlertRegistryAlert to Alert
• Util
  • util-core
    • Env
      • Remove newPrefixedThreadFactory and shutDownExecutor. Use ThreadFactoryBuilder and MoreExecutors.shutdownAndAwaitTermination directly
  • util-web
    • Replace request module used in web-build.js script with node-fetch
    • Switch to Bootstrap Icons
    • Add web.css resource
      • Includes import for bootstrap and icons. Import into application stylesheet with

        @import url("../lib/util-web/css/web.css");
        

    • Change build configurations to unpack Maven web modules to ${basedir}/src/main/resources/lib instead of ${project.build.directory}. Now all web resources are available for r.js and we can use shared resources from other modules
    • ext
      • Remove db property. Use keyval module directly
    • ui
      • In modal KO binding, remove confirmClose and add new modalStatic option. Now uses static backdrop that has the same effect
• Meta
  • meta-client
    • Remove request options parameter from all request/reply methods
    • Client
      • Fix CONNECTERD_PROPERTY spelling to CONNECTED_PROPERTY
  • meta-engine
    • EngineRuntime
      • Rename shutDownExecutor to shutdownAndAwaitTermination
    • EngineResources class added for common functions used in resource classes
      • verifyToken now returns unauthorized (401) or forbidden (403) responses, so TokenHelper does not get stuck in a loop of unauthorized requests
  • meta-test
    • Remove service name prefix from all public static final test variables
• Bootstrap
  • bootstrap-core
    • Add moduleDirWin variable for execution and make sure moduleDir is always a Linux style path

Config

• ProvisionServerConfig
  • Add deleteExisting, reboot options
  • ProvisionProviderVultrConfig
    • Add blockStorageId option

Other

• Remove package.json from JSDoc build
• Rename meta-engine-test module to meta-test, since it contains test cases for core, client and engine
• Moved ide to its own platform-ide repository
  • Run mvn -DunpackIde to build and unpack the latest IDE
  • bootstrap-dev-env now uses a snapshot version downloaded from Nexus

Model

• MetaLog
  • Renamed from MetaLogEntry
• MetaUser
  • OTP/2FA options are not in otp object with a MetaUserOtpType marking the active option
• MetaUserConfig
  • Has a id field, instead of using a computed key from user and group
• WatchdogStatus
  • Does no longer have checks and metrics. Instead use alerts to only report problems
• MetaSchema
  • New class for handling JSON schema related tasks
  • Field types are no longer dependent on a particular name (eg. field ending in dateTime). There are meta types now that can be used in JSON schemas
  • Only fields defined in the MetaType JSON schema are indexed in Lucene
  • 2019-09 JSON schemas are now loaded from local resources to make it work offline
• Model$Lucene
  • Now contains constants and helper methods for Lucene operations

Protocol

• MetaNotification
  • Rename entry to log
• ActionRequest
  • Rename params to param
• WriteAction
  • New PARTIAL_UPDATE value that replaces the boolean flag on write requests
• WatchdogUpdateRequest/WatchdogUpdateReply
  • Removed. Local updates now use the normal report mechanism with a special engine value self

    "watchdog": {
        "report": {
            "engine": "self",
            "engineId": "localhost",
            "password": null,
            "updatePassword": null
            "updateUser": null,
            "user": null,
        }
    }
    

• egn as a default channel does no longer exist. If no channel is specified (eg. in WriteRequest), then no notifications are dispatched

Config

• ClientConfig
  • Removed all web socket configuration options
• FileServiceConfig
  • Removed cleanupSchedule, bufferSize
• HttpServiceConfig
  • Removed all web socket configuration options
• LoggingServiceConfig
  • Removed monitorSchedule
• LuceneServiceConfig
  • Removed commitSchedule, workQueue
• WatchdogServiceConfig
  • Removed update config

Other

• Web
  • Merged into util
  • web-core is now util-web
  • web-core-test is now util-web-test
• Util
  • util-core
    • Json contains everything JSON
  • util-web
    • Update to Bootstrap 5
      • Use constraint API for form validation
      • Remove jQuery as a dependency
    • Replace Moment.js with Luxon
      • ext contains parsing/formatting functions
    • Add JSDoc integration
    • device
      • With the removal of Android wrapper, refactored to a static collection of browser functions
    • ui
      • Add toast KO binding
      • Add validity KO binding
• Meta
  • meta-core
    • Cst/cst modules were removed and their contents distributed to other modules
  • meta-client
    • viewmodel
      • New helper module for viewmodel related functions that were previously in meta-core/mode or other places
    • component/login
      • Contains all means to change OTP methods, like generating QR code for TOTP or the WebAuthn registration/login flow
      • meta-admin currently does not contain any of these functions to prevent duplication of code. Use login component to change OTP methods of an user (by generating a new password, if necessary)
  • meta-engine
    • ActionService
      • Expects services to register with registerExecuteDelegate
    • FileService
      • Expects services to register with registerBackupDelegate
    • MetaService
      • Add search method with native Lucene query and sort to allow for more complex search operations from other services
      • Partial MetaObject updates using buildObjectUpdate now work recursive on any depth
    • WatchdogService
      • Uses normal report mechanism for local updates, see protocol changes
    • Remove EngineUploadResource
  • meta-engine-test
    • Use Builder interfaces in Dagger components to set service configurations
    • Default user USER_SERVICE_TEST_USER is now plat with password plat
    • Importer is now FixtureImporter
    • See WebDriverTest in meta-admin-test for references on how to setup runtime, import test data and run UI tests
• Bootstrap
  • Rename framework/bootstrap/bootstrap-test to bootstrap-core-test to prevent name clash with bootstrap/bootstrap-test
  • bootstrap-core
    • JSON format changed
      • Old

        {
            "copy": {
                "source": "{moduleDir}/fixture/certbot-deploy.sh",
                "target": "{toolDir}/certbot/deploy.sh",
                "iif": {
                    "env": "NIX"
                }
            }
        }
        

      • New

        {
            "action": {
                "type": "copy",
                "source": "{moduleDir}/fixture/certbot-deploy.sh",
                "target": "{toolDir}/certbot/deploy.sh"
            },
            "condition": {
                "env": "NIX"
            }
        }
        

    • No longer uses command line arguments for -execute. All parameters are set in Execute configuration file. The bootstrap.json file is generated during build with -build (but can still be manually changed before execution if needed)
    • Desired version of the bootstrap module is now defined in Build configuration
      • This can be used to checkout the correct repository revision in automated build jobs with the new -buildversion argument
    • No longer use Maven filtering at build time. copy action now can filter a file and replace any variables in it with "filter": true
    • Rename binDir, customerDir, moduleDir used in Build configurations (outside of bootstrap.properties) to buildBinDir, buildCustomizeDir, buildModuleDir
  • bootstrap-common
    • engine-*.json definitions now use different directories
      • Engine {toolDir}/{serviceName}/engine
      • Web {toolDir}/{serviceName}/web
    • Rename dev-core.json definition to utils.json
    • New engine-server.json definition
• Android
  • Removed, as it is currently out of scope. A new solution should target both Android and iOS devices

0.761 - 0.1215

Released between 2018-12-03 and 2021-09-01.

See SCM history.

0.1 - 0.760

Released between 2015-06-02 and 2018-12-03.

See SCM history.

Pre 0.1

Sometime after 2013.

Cheatsheet