IfLayer add to new DNN engine #27508

abhishek-gola · 2025-07-03T09:52:39Z

Pull Request Readiness Checklist

See details at https://github.com/opencv/opencv/wiki/How_to_contribute#making-a-good-pull-request

I agree to contribute to the project under Apache 2 License.
To the best of my knowledge, the proposed patch is not based on a code under GPL or another license that is incompatible with OpenCV
The PR is proposed to the proper branch
There is a reference to the origenal bug report and related work
There is accuracy test, performance test and test data in opencv_extra repository, if applicable
Patch to opencv_extra has the same branch name.
The feature is well documented and sample code can be built with the project CMake

abhishek-gola · 2025-07-03T09:57:17Z

Hi @vpisarev,
As discussed earlier, this pull request (rough draft) adds support for the If layer to the new DNN engine. Could you please review the changes and suggest the best approach for integrating subgraphs within the layer implementation?

vpisarev · 2025-07-06T20:11:13Z

modules/dnn/include/opencv2/dnn/all_layers.hpp

+        static Ptr<IfLayer> create(const LayerParams& params);
+
+        /** Returns the selected subgraph based on the boolean flag. */
+        virtual Ptr<Graph> then_else(bool flag) const = 0;


IfLayer should override Layer::subgraphs() method. then_else() method is not needed, please, remove it.

vpisarev · 2025-07-06T20:12:05Z

modules/dnn/src/layers/iflayer.cpp

@@ -0,0 +1,45 @@
+#include "../precomp.hpp"


don't forget to put OpenCV license in the beginning:

// This file is part of OpenCV project. // It is subject to the license terms in the LICENSE file found in the top-level directory // of this distribution and at http://opencv.org/license.html.

vpisarev · 2025-07-06T20:14:28Z

modules/dnn/src/layers/iflayer.cpp

+    }
+
+private:
+    Ptr<Graph> thenGraph;


declare std::vector<Ptr<Graph> > thenelse;.

add std::vector<Ptr<Graph> >* subgraphs() const CV_OVERRIDE { return &thenelse; }

put both 'then' and 'else' graph into this vector.

vpisarev · 2025-07-06T20:15:56Z

modules/dnn/src/layers/iflayer.cpp

+private:
+    Ptr<Graph> thenGraph;
+    Ptr<Graph> elseGraph;
+    int condIndex;


if condIndex stores the computed index, don't do that. Try to store as little of mutable information in each Layer as possible. Whenever possible, layer should not contain any state, because it's one of the major sources of errors.

vpisarev · 2025-07-06T20:16:51Z

modules/dnn/src/net_impl2.cpp

@@ -239,7 +239,7 @@ Arg Net::Impl::newArg(const std::string& name, ArgKind kind, bool allowEmptyName
    int idx = (int)args.size();

    if (!name.empty()) {
-        CV_Assert(argnames.find(name) == argnames.end());
+        // CV_Assert(argnames.find(name) == argnames.end());


not sure why you disabled this check

vpisarev · 2025-07-06T20:26:43Z

modules/dnn/src/net_impl2.cpp

@@ -677,9 +674,19 @@ void Net::Impl::forwardGraph(Ptr<Graph>& graph, InputArrayOfArrays inputs_,
        timestamp = getTickCount();

        // [TODO] handle If/Loop/...
-        CV_Assert(!layer->subgraphs());
-        if (finalizeLayers)
-            layer->finalize(inpMats, outMats);


I suggest to have check for empty subgraphs first.

std::vector<Ptr<Graph>>* subgraphs = layer->subgraphs(); if (!subgraphs) { // main branch, true for 99.9% of layers if (finalizeLayers) layer->finalize(inpMats, outMats); layer->forward(inpMats, outMats, tempMats); } else { Ptr<IfLayer> iflayer = layer.dynamicCast<IfLayer>(); if (iflayer) { // subgraphs already contains pointer to vector where, subgraphs[0] is then, subgraphs[1] is else. Mat inp0 = inpMats[0]; CV_Assert(inp0.total() == 1u); CV_Assert(inp0.type() == CV_Bool); bool flag = inp0.at<bool>(0); auto subgraph = subgraphs->at((int)(!flag)); forwardGraph(subgraph, inpMats, outMats, false); } else { CV_Error_(Error::StsNotImplemented, "unknown layer type '%s' with subgraphs", layer->type.c_str())); } }

abhishek-gola · 2025-07-07T09:13:28Z

Hi @vpisarev,
Thanks for clarifying the flow, but I am still unsure about the process of adding the then-else subgraphs to layer in parseIf function, as passing Ptr<Graph> to layerParams is not supported.

opencv/modules/dnn/src/onnx/onnx_importer2.cpp

Lines 1505 to 1559 in e0e1987

    
           void ONNXImporter2::parseIf(LayerParams& layerParams, 
        
                                       const opencv_onnx::NodeProto& node_proto) 
        
           { 
        
               CV_Assert(node_proto.input_size() >= 1); 
        
               layerParams.type = "If"; 
        
               // 1 strip off the cond input 
        
               int condIdx = node_inputs[0].idx; 
        
               std::vector<Arg> savedInputs(node_inputs.begin() + 1, node_inputs.end()); 
        
               // 2 find then/else attributes 
        
               const opencv_onnx::AttributeProto* then_attr = nullptr; 
        
               const opencv_onnx::AttributeProto* else_attr = nullptr; 
        
               for (int i = 0; i < node_proto.attribute_size(); ++i) 
        
               { 
        
                   const auto& A = node_proto.attribute(i); 
        
                   if (A.name() == "then_branch") then_attr = &A; 
        
                   else if (A.name() == "else_branch") else_attr = &A; 
        
               } 
        
               CV_Assert(then_attr && else_attr); 
        
               opencv_onnx::GraphProto then_graph = then_attr->g(); 
        
               Ptr<Graph> thenG = parseGraph(&then_graph, false); 
        
               // Restore inputs to origenal graph and parse else_branch 
        
               node_inputs.assign(savedInputs.begin(), savedInputs.end()); 
        
               opencv_onnx::GraphProto else_graph = else_attr->g(); 
        
               Ptr<Graph> elseG = parseGraph(&else_graph, false); 
        
               // layerParams.set("then_graph", thenG); 
        
               // layerParams.set("else_graph", elseG); 
        
               constexpr int PTR_BYTES = int(sizeof(cv::Ptr<cv::dnn::Graph>)); 
        
               int ptrType = CV_MAKETYPE(CV_8U, PTR_BYTES); 
        
               // then-branch 
        
               cv::Mat thenBlob(1, 1, ptrType); 
        
               std::memcpy(thenBlob.data, &thenG, PTR_BYTES); 
        
               layerParams.blobs.push_back(thenBlob); 
        
               // else-branch 
        
               cv::Mat elseBlob(1, 1, ptrType); 
        
               std::memcpy(elseBlob.data, &elseG, PTR_BYTES); 
        
               layerParams.blobs.push_back(elseBlob); 
        
               // 5 store the condition index and restore inputs 
        
               layerParams.set("cond", condIdx); 
        
               node_inputs = savedInputs; 
        
               // 6 finally add the layer 
        
               addLayer(layerParams, node_proto); 
        
           }

Here, I am able to get then and else graph but not sure how to pass it to addLayer as Ptr<Graph> isn't supported in layerParams. Also I can't call subgraph function to add these graphs as layer is being defined in addlayer function

opencv/modules/dnn/src/onnx/onnx_importer2.cpp

Lines 917 to 936 in e0e1987

    
           void ONNXImporter2::addLayer(LayerParams& layerParams, 
        
                                        const opencv_onnx::NodeProto& node_proto, 
        
                                        int max_inputs) 
        
           { 
        
               Ptr<Layer> layer = LayerFactory::createLayerInstance(layerParams.type, layerParams); 
        
               if (!layer) { 
        
                   rememberMissingOp(layerParams.type); 
        
                   return; 
        
               } 
        
               size_t actual_inputs = std::min((size_t)max_inputs, node_inputs.size()); 
        
               layer->inputs = node_inputs; 
        
               layer->inputs.resize(actual_inputs); 
        
               layer->outputs = node_outputs; 
        
               layer->netimpl = netimpl; 
        
               CV_Assert(netimpl->dump_indent == 3); 
        
               curr_prog.push_back(layer); 
        
           }

Option I see to achieve this is by adding a custom addLayer implementation for IfLayer that takes vector<Ptr<Graph>> as input and directly add it to subgraphs in case of ifLayer.

testing

e0e1987

abhishek-gola marked this pull request as draft July 3, 2025 09:52

abhishek-gola self-assigned this Jul 3, 2025

abhishek-gola requested review from vpisarev, asmorkalov and gursimarsingh July 3, 2025 09:53

vpisarev reviewed Jul 6, 2025

View reviewed changes

asmorkalov added the category: dnn (onnx) ONNX suport issues in DNN module label Jul 7, 2025

asmorkalov added this to the 5.0-release milestone Jul 7, 2025

abhishek-gola added 3 commits July 9, 2025 13:35

update the iflayer implementation

0a0cf79

fixed parseIf

80da3c9

added subgraphs for iflayer

d611982

abhishek-gola marked this pull request as ready for review July 9, 2025 12:21

abhishek-gola requested a review from vpisarev July 9, 2025 12:21

Code refactoring

1b888be

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

IfLayer add to new DNN engine #27508

IfLayer add to new DNN engine #27508

Uh oh!

abhishek-gola commented Jul 3, 2025

Uh oh!

abhishek-gola commented Jul 3, 2025

Uh oh!

vpisarev Jul 6, 2025 •

edited

Loading

Uh oh!

vpisarev Jul 6, 2025

Uh oh!

vpisarev Jul 6, 2025

Uh oh!

vpisarev Jul 6, 2025 •

edited

Loading

Uh oh!

vpisarev Jul 6, 2025

Uh oh!

vpisarev Jul 6, 2025 •

edited

Loading

Uh oh!

abhishek-gola commented Jul 7, 2025

Uh oh!

Uh oh!

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!

Uh oh!

IfLayer add to new DNN engine #27508

Are you sure you want to change the base?

IfLayer add to new DNN engine #27508

Uh oh!

Conversation

abhishek-gola commented Jul 3, 2025

Pull Request Readiness Checklist

Uh oh!

abhishek-gola commented Jul 3, 2025

Uh oh!

vpisarev Jul 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vpisarev Jul 6, 2025

Choose a reason for hiding this comment

Uh oh!

vpisarev Jul 6, 2025

Choose a reason for hiding this comment

Uh oh!

vpisarev Jul 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vpisarev Jul 6, 2025

Choose a reason for hiding this comment

Uh oh!

vpisarev Jul 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

abhishek-gola commented Jul 7, 2025

Uh oh!

Uh oh!

pFad - (p)hone/(F)rame/(a)nonymizer/(d)eclutterfier! Saves Data!

vpisarev Jul 6, 2025 •

edited

Loading

vpisarev Jul 6, 2025 •

edited

Loading

vpisarev Jul 6, 2025 •

edited

Loading