This page contains information about NETCONF (Scale, Performance etc.) testing.
Scale tests for NETCONF in ODL.
Goal of this test is to measure how many NETCONF devices can be mounted by ODL with a set amount of RAM.
java -Xmx8G -jar scale-util-1.1.0-SNAPSHOT-scale-util.jar --distribution-folder ./distribution-karaf-0.4.0-Beryllium --device-count 8000 --ssh false --exi false --generate-configs-batch-size 1000
The scale util needs to be pointed to an unpacked distribution (--distribution-folder argument) and handles the karaf start and feature installation. While the test is running the utility is also checking Restconf periodically to see the current status of all netconf devices. After the test completes successfully karaf is stopped, features are cleaned and the whole test is restarted with more devices(currently hardcoded 1000, parameter for this needs to be added). If you are starting with more ram than 2GB you should start with more devices than 8k.right away. The test results are being logged periodically into the scale-results.log that's present in the same location as the util jar. If you are running with even more devices the ram for testtools should be increased aswell.
To run the test with tcp add --ssh false argument when starting the scale-util.
To rerun the test with more ram available to odl you need to edit the /${distribution-location}/bin/setenv script line:
export JAVA_MAX_MEM="4G"
NOTE: the fastest way to find out how many devices odl can handle at a given ram is to start the test with a larger amount of devices than it can handle, set the config batch size to 1k and start the test. You can then analyze the result log to see where there seems to be a drop off in the connected devices/or the test timed out.
Environment:
Other configuration:
In this test the simulated devices testtool started only had the base capabilites that testtool has included by default(ietf-netconf-monitoring, ietf-inet-types, ietf-yang-types).
Netconf scale test | |||
---|---|---|---|
Ram for ODL - 2GB | |||
transport | devices | time needed | |
ssh | 8000 | 3m 40s | 4k batches, starts having low memory(most of the time in GC) issues around 8k devices. |
ssh | 9000 | 12m 16s | 1k batches, times out after 20minutes |
tcp | 20000 | 6m 03s | 4k batches, reached 20min timeout, maybe can handle a bit more |
tcp | 21000 | 18m 54s | 1k batches, reached 20min timeout, maybe can handle a bit more |
Ram for ODL - 4GB | |||
ssh | 14000 | 9m 28s | |
ssh | 15000 | 17m 20s | timeout after 20minutes, hits the ram limit |
tcp | 24000 | 18m 31s | 1k batches |
tcp | 28000 | 17m 27s | 2k batches, timeout after 20min, should be able to get higher but needs more time |
With tcp we also noticed that after 15k devices theres a pretty big slowdown with pushing/handling the configs as there starts to be increasing gaps between the individual batches.
Environment:
Other configurations
Howto
Git clone https://github.com/opendaylight/netconf.git cd netconf mvn clean install cp ~/netconf/netconf/tools/netconf-testtool/target/scale-util-1.2.0-SNAPSHOT.jar ~
java -Xmx8G -jar scale-util-1.2.0-SNAPSHOT-scale-util.jar --distribution-folder ./distribution-karaf-0.4.0-Beryllium --device-count 8000 --ssh false --exi false --generate-configs-batch-size 1000
Results:
The Netconf Southbound test would repeatedly fail with a "java.lang.OutOfMemoryError: unable to create a new native thread" error message. The tester used the command set defined above, although the flag -Xmx8G was decreased to 4G, increased to 16G, and 32G -- all with the same result. At the same time this tester increased the value of JAVA_MAX_MEM to 16G and 32G, and still encountered the java.lang.OutOfMemoryError message. Log files of the failure and hs_err_pid log files captured and saved for examination. These are available on request.
Performance tests for NETCONF in ODL.
Goal of this test is to measure is the performance of an external NETCONF client uploading information into ODL (Global Datastore) using just NETCONF northbound server.
This how to will be split into multiple ordered sections:
git clone https://git.opendaylight.org/gerrit/coretutorials cd coretutorials git checkout stable/beryllium git fetch https://git.opendaylight.org/gerrit/coretutorials refs/changes/16/35916/1 && git checkout FETCH_HEAD cd ncmount mvn clean install -DskipTests -Dcheckstyle.skip cd karaf/target/assembly/
./bin/karaf
NETCONF Node: controller-config is fully connected
curl -u "admin:admin" -H "Accept: application/xml" -H "Content-type: application/xml" --request POST 'http://localhost:8181/restconf/config/network-topology:network-topology/topology/topology-netconf/node/controller-config/yang-ext:mount/config:modules' --data '<module xmlns="urn:opendaylight:params:xml:ns:yang:controller:config"> \ <type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:netconf:northbound:tcp">prefix:netconf-northbound-tcp</type> \ <name>netconf-mdsal-tcp-server</name> \ <dispatcher xmlns="urn:opendaylight:params:xml:ns:yang:controller:netconf:northbound:tcp"> \ <type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:config:netconf:northbound">prefix:netconf-server-dispatcher</type> \ <name>netconf-mdsal-server-dispatcher</name> \ </dispatcher> \ </module>'
Netconf TCP endpoint started successfully at /0.0.0.0:2831
edit-l2fib-1000.txt edit-l2fib-1.txt netconf-north-perf-test-files.zip stress-client-1.0.0-Beryllium-package edit-l2fib-100.txt edit-l2fib-delete-all.txt netconf-testtool-1.0.0-Beryllium-stress-client.tar.gz
java -Xmx2G -XX:MaxPermSize=256M -jar stress-client-1.0.0-Beryllium-package/stress-client-1.0.0-Beryllium-stress-client.jar --ip 127.0.0.1 --port 2831 --edits 10000 --exi false --ssh false --username admin --password admin --thread-amount 1 --async false --edit-batch-size 10000 --edit-content edit-l2fib-1.txt
FINISHED. Execution time: 5.585 s Requests per second: 1790.8309455587394
curl -u "admin:admin" http://localhost:8181/restconf/config/ncmount-l2fib:bridge-domains | grep -o forward | wc -l
java -jar stress-client-1.0.0-Beryllium-package/stress-client-1.0.0-Beryllium-stress-client.jar --ip 127.0.0.1 --port 2831 --edits 1 --exi false --ssh false --username admin --password admin --thread-amount 1 --async false --edit-content edit-l2fib-delete-all.txt
Note: The client has many configuration options. Use -h to see all of them. Note: There are 3 edit-content files in the resources. Each contains a different number of l2-fibs per edit-config: 1, 100 and 1000. Files with different amounts can be produced and used.
java -Xmx2G -XX:MaxPermSize=256M -jar stress-client-1.0.0-Beryllium-package/stress-client-1.0.0-Beryllium-stress-client.jar --ip 127.0.0.1 --port 2830 --edits 10000 --exi false --ssh true --username admin --password admin --thread-amount 1 --async false --edit-batch-size 10000 --edit-content edit-l2fib-1.txt
FINISHED. Execution time: 11.64 s Requests per second: 859.106529209622
Environment:
Base configuration:
Measured numbers with a single client:
Netconf northbound single client performance | ||||||
---|---|---|---|---|---|---|
Client type | l2fib per request | TCP performance | SSH performance | Total l2fibs | ||
Sync | 1 | 1 730 edits/s 1 730 l2fibs/s | 1 474 edits/s 1 474 l2fibs/s | 100k | ||
Async | 1 | 7 063 edits/s 7 063 l2fibs/s | 6 600 edits/s 6 600 l2fibs/s | 100k | ||
Sync | 100 | 233 edits/s 23 372 l2fibs/s | 148 edits/s 14 850 l2fibs/s | 500k | ||
Async | 100 | 421 edits/s 42 179 l2fibs/s | 386 edits/s 38 600 l2fibs/s | 500k | ||
Sync | 500 | 61 edits/s 30 935 l2fibs/s | 13 edits/s 6 590 l2fibs/s | 1M | ||
Async | 500 | 81 edits/s 40 894 l2fibs/s | 69 edits/s 34 500 l2fibs/s | 1M | ||
Sync | 1000 | 35 edits/s 35 365 l2fibs/s | 13 edits/s 13 248 l2fibs/s | 1M | ||
Async | 1000 | 38 edits/s 38 099 l2fibs/s | 19 edits/s 19 898 l2fibs/s | 1M |
Multiple clients:
Netconf northbound mutliplce client performance | |||||||
---|---|---|---|---|---|---|---|
Clients | Client type | l2fib per request | TCP performance | SSH performance | Total l2fibs | ||
8 | Sync | 1 | 23 010 edits/s 23 010 l2fibs/s | 13 847 edits/s 13 847 l2fibs/s | 400k | ||
8 | Async | 1 | 41 114 edits/s 41 114 l2fibs/s | 12 527 edits/s 12 527 l2fibs/s | 400k | ||
16 | Sync | 1 | 31 743 edits/s 31 743 l2fibs/s | 15 879 edits/s 15 879 l2fibs/s | 400k | ||
16 | Async | 1 | 43 252 edits/s 43 252 l2fibs/s | 12 496 edits/s 12 496 l2fibs/s | 400k | ||
8 | Sync | 100 | 852 edits/s 85 215 l2fibs/s | 769 edits/s 76 989 l2fibs/s | 1,6M | ||
8 | Async | 100 | 984 edits/s 98 419 l2fibs/s | 869 edits/s 86 923 l2fibs/s | 1,6M | ||
16 | Sync | 100 | 808 edits/s 80 885 l2fibs/s | 723 edits/s 72 345 l2fibs/s | 1,6M | ||
16 | Async | 100 | 852 edits/s 85 224 l2fibs/s | 749 edits/s 74 962 l2fibs/s | 1,6M | ||
8 | Sync | 500 | |||||
8 | Async | 500 | |||||
16 | Sync | 500 | |||||
16 | Async | 500 | |||||
8 | Sync | 1000 | |||||
8 | Async | 1000 | |||||
16 | Sync | 1000 | |||||
16 | Async | 1000 |
Environment
Other configurations
N/A
Steps to recreate
git clone https://git.opendaylight.org/gerrit/coretutorials cd coretutorials git checkout master cd ncmount mvn clean install –DskipTests –Dcheckstyle.skip cd karaf/target/assembly
(When the other set of instructions were followed, when the tester executed "mvn clean install –DskipTests –Dcheckstyle.skip", maven failed due to "Non-resolvable parent POM for org.opendaylight.coretutorials:ncmount-aggregator:1.1.0-SNAPSHOT". Attempting to hand-edit all of the relevant pom.xml files proved impractical -- too many files to investigate & fix -- so the tester used this work around. At least maven completed the command.)
mvn clean install
(Trying to download the Beryllium version of the NETCONF stress tool from the URL in the Wiki failed. When the tester used the link, he encountered the error “404 Not found: repository with ID: “autorelease-1074” not found”.)
./bin/karaf
At this point, the expected message "NETCONF Node: controller-config is fully connected" was not found in the logs. The tester was not confident he understood these instructions, & stopped here, waiting on further instructions.
Goal of this test is to measure is the performance of a NETCONF device uploading information into ODL using just NETCONF southbound notifications.
Notes:
java -jar netconf-testtool-1.0.0-Beryllium-executable.jar --schemas-dir ./xrSchemas/ --ssh false --exi false --notification-file i2rs-notifs-perf100k.xml
./ncmount-karaf-1.1.0-SNAPSHOT/bin/karaf
and wait until you there's following message in the log:
NETCONF Node: controller-config is fully connected
curl -u "admin:admin" -H "Accept: application/xml" -H "Content-type: application/xml" --request POST 'http://localhost:8181/restconf/config/network-topology:network-topology/topology/topology-netconf/node/controller-config/yang-ext:mount/config:modules' --data '<module xmlns="urn:opendaylight:params:xml:ns:yang:controller:config"> \ <type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">prefix:sal-netconf-connector</type> \ <name>controller-notif-100000</name> \ <address xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">127.0.0.1</address> \ <port xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">17830</port> \ <username xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">admin</username> \ <password xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">admin</password> \ <tcp-only xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">true</tcp-only> \ <keepalive-delay xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf">0</keepalive-delay> \ <event-executor xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"> \ <type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:netty">prefix:netty-event-executor</type> \ <name>global-event-executor</name> \ </event-executor> \ <binding-registry xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"> \ <type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:md:sal:binding">prefix:binding-broker-osgi-registry</type> \ <name>binding-osgi-broker</name> \ </binding-registry> \ <dom-registry xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"> \ <type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:md:sal:dom">prefix:dom-broker-osgi-registry</type> \ <name>dom-broker</name> \ </dom-registry> \ <client-dispatcher xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"> \ <type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:config:netconf">prefix:netconf-client-dispatcher</type> \ <name>global-netconf-dispatcher</name> \ </client-dispatcher> \ <processing-executor xmlns="urn:opendaylight:params:xml:ns:yang:controller:md:sal:connector:netconf"> \ <type xmlns:prefix="urn:opendaylight:params:xml:ns:yang:controller:threadpool">prefix:threadpool</type> \ <name>global-netconf-processing-executor</name> \ </processing-executor> \ </module>'
NETCONF Node: controller-notif-100000 is fully connected
Elapsed ms for 100000 notifications: 9092 Performance (notifications/second): 10998.680158380994 Performance (prefixes/second): 10998.680158380994
curl -u "admin:admin" -H "Accept: application/xml" -H "Content-type: application/xml" --request DELETE http://localhost:8181/restconf/config/network-topology:network-topology/topology/topology-netconf/node/controller-config/yang-ext:mount/config:modules/module/odl-sal-netconf-connector-cfg:sal-netconf-connector/controller-notif-100000
Notes:
Environment:
Configuration:
Measured performance send/receive of 1M Notifications:
Netconf southbound notification performance | |||||||
---|---|---|---|---|---|---|---|
Total notifications | Prefixes per notification | TCP performance | SSH performance | TCP+EXI performance | |||
100k | 1 | 10716 notifications/s 10716 prefixes/s | 9828 notifications/s 9828 prefixes/s | ||||
100k | 2 | 7112 notifications/s 14224 prefixes/s | 5496 notifications/s 10992 prefixes/s | ||||
100k | 10 | 1996 notifications/s 19965 prefixes/s | 1635 notifications/s 16356 prefixes/s |
* SSH test performance is worse over time (with test reruns on the same ODL) and more memory is used but not freed. Looks like a memory leak. https://bugs.opendaylight.org/show_bug.cgi?id=5488
Test failed to run.
Environment
Steps performed
Goal of this test is to measure is the performance the end-to-end (external REST client -> REST north -> MD-SAL -> NETCONF south -> NETCONF device) performance of ODL using both NETCONF and RESTCONF.
git clone https://git.opendaylight.org/gerrit/coretutorials cd coretutorials git fetch https://git.opendaylight.org/gerrit/coretutorials refs/changes/54/36054/1 && git checkout FETCH_HEAD cd ncmount mvn clean install -DskipTests -Dcheckstyle.skip cd karaf/target/assembly/
java -jar netconf-testtool-1.0.0-Beryllium-executable.jar --schemas-dir ./xrSchemas/ --ssh false --exi false --distribution-folder /ncmount-karaf-1.1.0-SNAPSHOT
./ncmount-karaf-1.1.0-SNAPSHOT/bin/karaf
java -jar rest-stress-client.jar --ip localhost --port 8181 --destination /restconf/operations/ncmount:write-routes --edits 100 --edit-content json_routes_10.json --async-requests true --throttle 1000 --auth admin admin
To run this test with multiple devices/clients these changes apply:
--device-count 16
--same-device false --thread-amount 16
--thread-amount is the number of clients. Each client will be mapped to a single device.