Proposal 1 - Direct OpenFlowJava (OFJ) models use

Changes on bulk-o-matic enables to test using OFJ model to test performance improvements. Using those models gives should give us possibility to avoid translators and spare time and memory usage. This changes are related to the first changes proposal from  this document.

Change of the service

In SalFlowService in OpenFlowPlugin (OFP) were added new RPC

rpc add-flow-direct {
    description "Adding flow to openflow device.";
    input {
        uses tr:transaction-metadata;
        leaf flow-ref {
            type types:flow-ref;
        }
        uses node-flow-direct;
    }
    output {
        uses tr:transaction-aware;
    }
}

This RPC uses the openflow protocol specification flow.

Model comparison

OFP standard flow

AddFlowInput[
   _flowRef=FlowRef [
      _value=KeyedInstanceIdentifier {
         targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.tables.table.Flow,
         path= [
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes,
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node [
               key=NodeKey [
                  _id=Uri [
                     _value=openflow:3
                  ]
               ]
            ],
            org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableNode,
            org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.tables.Table [
               key=TableKey [
                  _id=1
               ]
            ],
            org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.tables.table.Flow [
               key=FlowKey [
                  _id=Uri [
                     _value=600
                  ]
               ]
            ]
         ]
      }
   ],
   _flowTable=FlowTableRef [
      _value=KeyedInstanceIdentifier {
         targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.tables.Table,
         path= [
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes,
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node [
               key=NodeKey [
                  _id=Uri [
                     _value=openflow:3
                  ]
               ]
            ],
            org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableNode,
            org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.tables.Table [
               key=TableKey [
                  _id=1
               ]
            ]
         ]
      }
   ],
   _match=Match [
      _ethernetMatch=EthernetMatch [
         _ethernetType=EthernetType [
            _type=EtherType [
               _value=2048
            ],
            augmentation= []
         ],
         augmentation= []
      ],
      _layer3Match=Ipv4Match [
         _ipv4Source=Ipv4Prefix [
            _value=0.0.0.100/32
         ],
         augmentation= []
      ],
      augmentation= []
   ],
   _node=NodeRef [
      _value=KeyedInstanceIdentifier {
         targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node,
         path= [
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes,
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node [
               key=NodeKey [
                  _id=Uri [
                     _value=openflow:3
                  ]
               ]
            ]
         ]
      }
   ],
   _tableId=1,
   augmentation= []
]

OFJ specification flow

 AddFlowDirectInput [
   _bufferId=4294967295,
   _command=OFPFCADD,
   _cookie=0,
   _cookieMask=0,
   _flags=FlowModFlags [
      _oFPFFSENDFLOWREM=false,
      _oFPFFCHECKOVERLAP=false,
      _oFPFFRESETCOUNTS=false,
      _oFPFFNOPKTCOUNTS=false,
      _oFPFFNOBYTCOUNTS=false
   ],
   _flowRef=FlowRef [
      _value=KeyedInstanceIdentifier {
         targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.tables.table.Flow,
         path= [
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes,
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node [
               key=NodeKey [
                  _id=Uri [
                     _value=openflow:1
                  ]
               ]
            ],
            org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableNode,
            org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.tables.Table [
               key=TableKey [
                  _id=1
               ]
            ],
            org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.tables.table.Flow [
               key=FlowKey [
                  _id=Uri [
                     _value=600
                  ]
               ]
            ]
         ]
      }
   ],
   _flowTable=FlowTableRef [
      _value=KeyedInstanceIdentifier {
         targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.tables.Table,
         path= [
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes,
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node [
               key=NodeKey [
                  _id=Uri [
                     _value=openflow:1
                  ]
               ]
            ],
            org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.FlowCapableNode,
            org.opendaylight.yang.gen.v1.urn.opendaylight.flow.inventory.rev130819.tables.Table [
               key=TableKey [
                  _id=1
               ]
            ]
         ]
      }
   ],
   _hardTimeout=0,
   _idleTimeout=0,
   _match=Match [
      _matchEntry= [
         MatchEntry [
            _matchEntryValue=Ipv4SrcCase [
               _ipv4Src=Ipv4Src [
                  _ipv4Address=Ipv4Address [
                     _value=0.0.0.100
                  ],
                  augmentation= []
               ],
               augmentation= []
            ],
            _oxmClass=class org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.OpenflowBasicClass,
            _oxmMatchField=class org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.Ipv4Src,
            _hasMask=false,
            augmentation= []
         ],
         MatchEntry [
            _matchEntryValue=EthTypeCase [
               _ethType=EthType [
                  _ethType=EtherType [
                     _value=2048
                  ],
                  augmentation= []
               ],
               augmentation= []
            ],
            _oxmClass=class org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.OpenflowBasicClass,
            _oxmMatchField=class org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.EthType,
            _hasMask=false,
            augmentation= []
         ]
      ],
      _type=class org.opendaylight.yang.gen.v1.urn.opendaylight.openflow.oxm.rev150225.OxmMatchType,
      augmentation= []
   ],
   _node=NodeRef [
      _value=KeyedInstanceIdentifier {
         targetType=interface org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node,
         path= [
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.Nodes,
            org.opendaylight.yang.gen.v1.urn.opendaylight.inventory.rev130819.nodes.Node [
               key=NodeKey [
                  _id=Uri [
                     _value=openflow:1
                  ]
               ]
            ]
         ]
      }
   ],
   _outGroup=4294967295,
   _outPort=PortNumber [
      _value=4294967295
   ],
   _priority=32768,
   _tableId=TableId [
      _value=1
   ],
   _version=4,
   augmentation= []
]

Proposal 2 - Convert OpenflowPlugin models directly to raw bytes

Adding new OFJ serializers for OFP models is another way to avoid translators and spare time and memory usage. This changes are related to the second changes proposal from  this document. This proposal is easier to adapt for downstream projects, because it do not requires change of models, but it is harder to implement than first one, because making custom serializers and deserializers for OFP models will take a lot more time, than just switching models in Proposal 1.

Change of the service

In SalFlowService in OpenFlowPlugin (OFP) were added new RPC

rpc add-flow-raw {
    description "Adding flow to openflow device.";
    input {
        uses tr:transaction-metadata;
        leaf flow-ref {
            type types:flow-ref;
        }

        uses node-flow;
    }
    output {
        uses tr:transaction-aware;
    }
}

This RPC is exactly same as add-flow rpc, but it serves as path delimiter (if this rpc is called, we will skip translations and send openflowplugin models directly to openflowjava, so it can use our custom serializers).

Test results

Results for adding 5000 flows in 20 runs for each flow path with more advanced flow what have flow flags, instructions with actions and match with ipv4 source and destination.

Path types:

  • NORMAL: Uses current path, translating openflowplugin models to openflowjava models and then to raw bytes
  • DIRECT: Uses openflowjava models directly, translating them to raw bytes
  • RAW: Uses openflowplugin models and translates them directly to raw bytes

Results:


NORMALDIRECTRAW

Runs

  run # dur
  1     2258    (warmup)
  2     872
  3     1163
  4     610
  5     561
  6     590
  7     643
  8     395
  9     591
  10    294
  11    731
  12    630
  13    666
  14    549
  15    575
  16    643
  17    634
  18    537
  19    572
  20    658

  run # dur
  1     417     (warmup)
  2     764
  3     541
  4     556
  5     618
  6     647
  7     520
  8     622
  9     549
  10    626
  11    518
  12    523
  13    559
  14    592
  15    585
  16    531
  17    576
  18    428
  19    519
  20    925

  run # dur
  1     615     (warmup)
  2     614
  3     598
  4     683
  5     546
  6     622
  7     591
  8     639
  9     525
  10    592
  11    650
  12    616
  13    487
  14    245
  15    336
  16    552
  17    501
  18    546
  19    615
  20    388

Total

11914

11199

10346

Average

627

589

544

All numbers above are in milliseconds, and numbers in each run is duration between start of bulk-o-matic add flow job and successful flow add response from device for last flow in bulk-o-matic add flow job. So, for example when we are adding 5000 flows, we send RPC to bulk-o-matic, that will prepare these 5000 flows, and then sends each of them as RPC to SalFlowService. Last flow in this case will be flow with id 5500, because in bulk-o-matic, starting flow id is always 500. So, in this case, it will be duration between starting of sending these 5000 flows to SalFlowService and successful add flow response from device for flow with id 5500. We are getting all these informations from logs.

How to setup and run test

Testing script can be downloaded  here  and run on this  patch from Gerrit (important, make sure you are on latest patchset).

./flowpathtest.sh -h

Usage:
        -h               show this message
        -s               skip adding flows, only process logs
        -t <flow_type>   flow type used for performance tests, can be NORMAL/DIRECT/RAW (NORMAL)
        -p <log_path>    path to karaf log file (distribution/karaf/target/assembly/data/log/karaf.log)
        -f <num_flows>   total number of flows added per loop (1000)
        -r <num_runs>    how many runs should test do, higher is better for accuracy (10)
        -c <address>     address of controller (127.0.0.1:8181)
        -w <wait_time>   how many seconds should test wait between each run (30)

Example usage:

./flowpathtest.sh -f 5000 -r 50 -w 10 -c "127.0.0.1:8181" -t "NORMAL" -p "distribution/karaf/target/assembly/data/log/karaf.log"

Command above will try to add 5000 flows via Bulk-o-matic 50 times each with NORMAL path (OpenflowPlugin models -> translators -> device) with delay of 10 seconds between each run to prevent interfering of runs between each other. Proper delay is very important, because if we do not set big enough delay, logs will be messed up and testing script will not be able to get time of last succesfully added flow to compute total duration.

This test requires to have INFO logging in FlowWriterDirectOFRpc and DEBUG logging in SalFlowServiceImpl and rest of logging on minimum, so your log4j configuration should look like this:

log4j.logger.ROOT=ERROR
log4j.logger.org.opendaylight=ERROR
log4j.logger.org.opendaylight.openflowplugin.impl.services.SalFlowServiceImpl=DEBUG
log4j.logger.org.opendaylight.openflowplugin.applications.bulk.o.matic.FlowWriterDirectOFRpc=INFO
  • No labels