|Infrastructure Utilities Facts|
Project Creation Date: March 17, 2016
IRC: freenode.net #opendaylight-infrautils
- 1 Introduction
- 2 Features
- 2.1 Completed & Available
- 2.2 Planned Features (WIP)
- 2.3 Future Ideas
- 2.4 Removed Features
- 3 Appendix
Welcome to the Infrastructure Utilities project page!
This project offers technical utilities and infrastructures for other projects to use.
We currently (as of May 2017) consider infrautils a low-level offset 0 project. As is now, it only depends on odlparent, and does not have dependencies to e.g. controller & mdsal or yangtools. This is intentional, as this would allow such projects to use utilities from infrautils (without causing circular dependencies among offset 0 projects).
We anticipate that e.g. the upcoming ready service may eventually want to be used by other offset 0 projects, and therefore want to be careful to not accidentally introduce dependencies from infrautils to other offset 0 projects.
Should there be a need to build utilities requiring such dependencies in the future (for example Cache layer for mdsal, YANG RPC for ready, etc.), we should discuss whether to create a new infrautils2 (?) project with dependencies to infrautils and other offset 0 projects, or put such utilities into existing projects such as e.g. genius.
This presentation given at the OpenDaylight Oxygen Developer Forum in October 2017 gives a good overview of the project.
Completed & Available
In chronological order when merged to master:
Create, update and output counters is a basic tool for debugging and generating statistics in any system. We've developed a counter infrastructure integrated into ODL which has already been successfully used with multiple products, and more recently in debugging and fixing the OpenFlow plugin/Java and LACP modules.
Utils incl. org.opendaylight.infrautils.utils.concurrent
Bunch of small (non test related) low level general utility classes à la Apache (Lang) Commons or Guava and similar incl. utils.concurrent:
- ListenableFutures toCompletionStage & addErrorLogging
- CompletableFutures completedExceptionally
- CompletionStages completedExceptionally
- LoggingRejectedExecutionHandler, LoggingThreadUncaughtExceptionHandler, ThreadFactoryProvider, Executors newSingleThreadExecutor
also includes the (as of 2018.09 WIP) ExecutionOrigin & Co. in infrautils.utils.mdc.
- LogRule which logs (using slf4j-api) the start and stop of each @Test method
- LogCaptureRule which can fail a test if something logs any ERROR from anywhere (This work even if the LOG.error() is happening in a background thread, not the test's main thread... which can be particularly interesting.)
- RunUntilFailureRule which allows to keep running tests indefinitely; for local usage to debug "flaky" (sometimes passing, sometimes failing) tests until they fail
- ClasspathHellDuplicatesCheckRule verifies, and guarantees future non-regression, against JAR hell due to duplicate classpath entries. Tests with this JUnit Rule will fail if their classpath contains duplicate class. This could be caused e.g. by changes to upstream transitive dependencies. See also http://jhades.github.io (which this internally uses, not exposed).
- see https://github.com/opendaylight/infrautils/blob/master/testutils/src/test/java/org/opendaylight/infrautils/testutils/tests/ExampleTest.java
Also some low level general utility classes helpful for unit testing concurrency related things in infrautils.testutils.concurrent:
- CompletionStageTestAwaiter, see CompletionStageAwaitExampleTest
JobCoordinator service which enables executing jobs in a parallel/sequential fashion based on their keys.
Originally in genius project, but moved to infrautils to be more widely available to other projects as well, as it was deemed to be a technical infrastructure not related to genius' project network abstraction. Subsequently improved within infrautils. Genius still has the previous incarnation of this code as deprecated util as of May 2017, but this is about to disappear via c/60303.
This was originally built for Daexim Import on Boot for Upgrade-ability, see https://git.opendaylight.org/gerrit/#/c/56686/ and https://git.opendaylight.org/gerrit/#/c/55035/. It is now also used by infrautils.diagstatus, which builds on top of infrautils.ready.
The implementation internally ultimately uses the same Karaf API that e.g. the standard "diag" Karaf CLI command uses. This checks both if all OSGi bundles have started as well if their blueprint initialization has been successfully fully completed. It builds on top of the bundles-test-lib from odlparent, which is what we run as SingleFeatureTest (SFT) during all of our builds to ensure that all projects' features can be installed without broken bundles.
What infrautils.ready adds on top of the underlying raw Karaf API is operator friendly logging, a convenient API and a correctly implemented polling loop in a background thread with SystemReadyListener registrations and notifications, instead of ODL applications re-implementing this. The infrautils.read intentionally API isolates consumers from the Karaf API. We encourage all ODL projects to use this infrautils.ready API instead of trying to reinvent the wheel and directly depending on the Karaf API, so that application code could be used outside of OSGi, in environment such as unit and component tests, or something such as honeycomb.
Applications can use this SystemReadyMonitor registerListener(SystemReadyListener) in a constructor to register a listener for and get notified when all bundles are "ready" in the technical sense (have been started in the OSGi sense and have completed their blueprint initialization), and could on that event do any further initialization it had to delay in the original blueprint initialization.
This cannot directly be used to express functional dependencies BETWEEN bundles (because that would deadlock infrautils.ready; it would stay in BOOTING forever and never reach SystemState ACTIVE). The natural way to make one bundle await another is to use Blueprint OSGi service dependency. If there is no technical service dependency but only a logical functional one, then in infrautils.ready.order there is a convenience sugar utility to publish "marker" FunctionalityReady interfaces to the OSGi service registry; unlike real services, these have no implementing code, but another bundle could depend on one to enforce start up order one (using regular bluepring <reference> in XML or @OsgiService annotation).
A known limitation of the current implementation of infrautils.ready is that its "wait until ready" loop runs only once, after installation of infrautils.ready (by boot feature usage, or initial single line feature:install). So SystemState will go from BOOTING to ACTIVE or FAILURE, once. So if you do more feature:install after a time gap, there won’t be any further state change notification; the currently implementation won't "go back" from ACTIVE to BOOTING. (It would be absolutely possible to extend SystemReadyListener onSystemBootReady() with an onSystemIsChanging() and onSystemReadyAgain(), but the original author has no need for this; as "hot" installing additional ODL application features during operational uptime was not a real world requirement to the original author. If this is important to you, then your contributions for extending this would certainly be welcome.)
Another known more minor issue in infrautils.ready is that its logging during boot is can confuse people, because its too verbose; this will be fixed by https://jira.opendaylight.org/browse/INFRAUTILS-17 via https://git.opendaylight.org/gerrit/#/c/66013/ when the odlparent 3.0.0 roll is completed (or if https://git.opendaylight.org/gerrit/#/c/66202/ is accepted into odlparent 2.0.x and released).
An enhancement to infrautils.ready requested by infrautils.diagstatus is https://jira.opendaylight.org/browse/INFRAUTILS-27 (likewise waiting for odlparent).
infrautils' ready, like other infrautils APIs, is available as a separate Karaf feature. Downstream projects using infrautils.ready will therefore NOT pull in other bundles for other infrautils functionalities.
Integration Test Utilities (itestutils)
Used for non-regression self-testing of features in this project (and available to others).
See https://www.youtube.com/watch?v=h4HOSRN2aFc and play with the example in infrautils/caches/sample installed by odl-infrautils-caches-sample; history in https://git.opendaylight.org/gerrit/#/c/48920/ and https://bugs.opendaylight.org/show_bug.cgi?id=8300.
Planned Features (WIP)
- based on http://metrics.dropwizard.io
- see https://git.opendaylight.org/gerrit/#/q/topic:metrics
- background slide https://codahale.com/codeconf-2011-04-09-metrics-metrics-everywhere.pdf
Logging with ExecutionOrigin ID
Linux Foundation (LF) Cloud Native Computing Foundation (CNCF) Open Tracing for OpenDaylight and its north- and southbound!
Remove in https://git.opendaylight.org/gerrit/#/c/53215/ : The decision to split a service into one or more threads with asynchronous interactions between them is frequently dependent on constraints learned late in the development and even the deployment cycle. In order to allow flexibility in making these decisions we've developed an infrastructure which is configuration driven allowing agnostic code to be written under generic constrains which can then later be customized according to the required constraints.
The infrautils project does not publish forward looking release plans. This Wiki page does however have a "Completed & Available" and a "Planned Features (WIP)" section above. So whatever is in is in by cut off and makes it, what's not gets deferred to the next release. Parties interested in "Planned Features (WIP)" are welcome to contribute!