14 Replies Latest reply: Oct 14, 2013 2:21 AM by ADVUNI-TO1 RSS

SnapCreator Plugin "Cache" SoapFault

ADVUNI-BN1 Novice
Currently Being Moderated

Sometimes we get the following error... after we restared the agent service everything work fine for a few days until it happens again.

 

Snap Creator Framework 4.0p1

### Snap Creator Framework 4.0p1

NetApp Release 8.1.2P4

 

########## Application Quiesce ##########

[2013-08-30 13:00:02,107] ERROR: SCF-00037: Application quiesce for plugin [CACHE] failed with error [org.apache.cxf.binding.soap.SoapFault: Can't call method "references" on an undefined value at /<C:\Program Files\NetApp\Snap_Creator_Framework\scAgent4.0p1\snapcreator.exe>SOAP/Lite.pm line 2799.

] and exit code [-1], Exiting!

 

 

Does anyone has an idea to resolv this issue?

 

Regards

Benny

  • Re: SnapCreator Plugin "Cache" SoapFault
    ktenzer NetApp Employee NASCAR racer
    Currently Being Moderated

    Looks like a SOAP communication issue, under certain conditions or at times it appears the CACHE plugin returns an invalid result type back to SC server. You can try with SC 3.6P1 as this was more lenient toward SOAP message formulation, however ultimately this appears to be a plugin issue.

     

    I will look into plugin and see if I can find anything

     

    Regards,

     

    Keith

    • Re: SnapCreator Plugin "Cache" SoapFault
      ADVUNI-BN1 Novice
      Currently Being Moderated

      Thank for reply!

       

      We use some more Plugins and did the Update from 3.6 to 4.0 a few weeks ago. At the moment its difficult to do the downgrade.

       

      Did you find something in the plugin?

       

      Regards,

      Benny

      • Re: SnapCreator Plugin "Cache" SoapFault
        Siva Ramanathan NetApp Employee Kart Racer
        Currently Being Moderated

        Hello Benny,

        Please email a scdump output to sivar at netapp.com

        I shall review the logs and get back to you asap.

         

        Thanks,

        Siva Ramanathan

        • Re: SnapCreator Plugin "Cache" SoapFault
          ADVUNI-TO1 Novice
          Currently Being Moderated

          Hi Siva Ramanathan,

           

          I have a similar error with the MAXDB plugin at another customer (first occurrence after customer upgraded to SC 4.0P1):

           

          [2013-08-28 10:35:34,790] ERROR: SCF-00037: Application quiesce for plugin [maxdb] failed with error [org.apache.cxf.binding.soap.SoapFault: Can't call method "references" on an undefined value at /<C:\Program Files\NetApp\Snap_Creator_Framework\scAgent4.0p1\snapcreator.exe>SOAP/Lite.pm line 2799.

           

          For me this looks like it is not a plugin issue but a general SOAP problem. I can mail you two SCDUMPs if you find that helpful. Just let me know if you'd like them.

           

          Regards,

          Tobias

        • Re: SnapCreator Plugin "Cache" SoapFault
          ADVUNI-TO1 Novice
          Currently Being Moderated

          Hi,

           

          do you have any update on this matter?

           

          Best regards

          Tobias

          • Re: SnapCreator Plugin "Cache" SoapFault
            Siva Ramanathan NetApp Employee Kart Racer
            Currently Being Moderated

            Tobias,

            I don't have an update yet.

            If you want me to review this with you over a webex session, I can schedule one for you.

            Please email me your availability next week to sivar at netapp.com

            Thanks,
            Siva Ramanathan

            • Re: SnapCreator Plugin "Cache" SoapFault
              ADVUNI-BN1 Novice
              Currently Being Moderated

              Hi,

               

              we have got now three issues at three customers with the same error:

              [2013-08-28 10:35:34,790] ERROR: SCF-00037: Application quiesce for plugin [maxdb] failed with error [org.apache.cxf.binding.soap.SoapFault: Can't call method "references" on an undefined value at /<C:\Program Files\NetApp\Snap_Creator_Framework\scAgent4.0p1\snapcreator.exe>SOAP/Lite.pm line 2799.

              ] and exit code [-1], Exiting!

               

              The error occurs at different plugins (Cache, maxDB) and always the agent chrashes.

               

              We need assistence...

               

              Could you find some further informations?

               

              Regards Benjamin

              • Re: SnapCreator Plugin "Cache" SoapFault
                ktenzer NetApp Employee NASCAR racer
                Currently Being Moderated

                BURT 761317 has been opened and this has been escalated so it is being looked at. We arent able yet to reproduce problem but hopefully soon we will and then we can provide a fix or make recommendation for workaround.

                 

                If you could provide all the environment details and steps to reproduce this error that would be helpful? Especially interesting would be if perl is installed on these systems and if so what version? Does this happen on only windows systems or unix? Is there some pattern here that can be recognized?

                 

                As for workaround: I heard from the BURT case that SC 3.6 P1 does not have this issue so if you are in a pinch and need to get something working asap I would go that route until this issue is resolved.

                 

                Regards,

                 

                Keith

              • Re: SnapCreator Plugin "Cache" SoapFault
                ktenzer NetApp Employee NASCAR racer
                Currently Being Moderated

                Oh and another thing to try I forgot to mention is set this in config:

                 

                PLUGIN_RESULT_FILTERING_ENABLE=Y

                 

                SC 4.0 server is using cxf which is very strict with SOAP message handling and the SC Agent is using Perl which isnt a strict. This option basically will ensure the Agent filters anything out of response that plugin might be adding which shouldnt be there.

                 

                I dont know that this will fix issue but please test this, it would help us tremendously to know.

                ?

                 

                Thanks!

                 

                Keith

                • Re: SnapCreator Plugin "Cache" SoapFault
                  ADVUNI-BN1 Novice
                  Currently Being Moderated

                  Thank for your answer!

                   

                  Our issues occure alway with Windows OS.

                  One customer tried PLUGIN_RESULT_FILTERING_ENABLE, but the issue even exists.

                   

                  I try to figure out things about perl...

                  • Re: SnapCreator Plugin "Cache" SoapFault
                    ktenzer NetApp Employee NASCAR racer
                    Currently Being Moderated

                    Ok I wasn't too hopeful that would resolve issue but it is helpful to know that issue is outside of plugin. It sounds on paper like a compatibility issue between perl that may be installed on system and the perl we ship with Snap Creator. However in 4.0 we use same version as 3.6 so this doesnt make sense. If 4.0 doesnt work than niether should 3.6.

                     

                    I am going to spend some more time trying to reproduce this tomorrow but I know where to focus at least...which is windows and maxdb.

                     

                    Regards,

                     

                    Keith

                  • Re: SnapCreator Plugin "Cache" SoapFault
                    ktenzer NetApp Employee NASCAR racer
                    Currently Being Moderated

                    Ok I have reproduced the problem.

                     

                    It appears to happen in windows 2008 environment R2 when running SC Agent as a service. All plugins are affected not just maxdb. I ran backups every minute for several hours and after a while you run into this issue. It only affects quiesce, other operations work strangely.

                     

                    We are still trying to pinpoint the exact problem but for now you can try disabling service and running SC agent manually: scAgent4.0.0/snapcreator.exe --start-agent 9090 --verbose --debug

                     

                    If this works you can setup a task manager job to run bat script which starts SC server manually upon system startup and runs it in background. Basically this is same as what service does.

                     

                    I will update this when we pinpoint problem and then we should have either an acceptable workaround or a planned patch / fix.

                     

                    Regards,

                     

                    Keith

                  • Re: SnapCreator Plugin "Cache" SoapFault
                    ktenzer NetApp Employee NASCAR racer
                    Currently Being Moderated

                    UPDATE:

                     

                    We have identified the problem. The watchdog functionality which monitors agent is not working in some windows environments properly, windows 2008 and possibly others. The watchdog process forks a process each time a quiesce request comes into agent and starts a timer, if no unquiesce comes in the watchdog will do a force unquiesce. This is to ensure if SC server crashes that database doesnt stay in a backup state.

                     

                    For windows encironments experiencing this issues please disable watchdog and things will work, you can do that by setting following in config:

                    SC_AGENT_WATCHDOG_ENABLE=N

                     

                    This issue will be resolved in SC 4.1 with the new agent. As for SC 4.0 we are not going to fix this issue. We may release a patch which will disable watchdog for windows and throw friendly message if you try and enable it. Watchdog is not critical to agent operation and doesnt solve problem if agent crashes and database is in backup mode.

                     

                    In SC 4.1 we have redesigned the agent and not only do we handle case where quiesce comes but no unquiesce happens due to SC server or network issue but also if agent crashes. The 4.1 agent will be much more robust and 4.1 will be available soon as a community release and beginning of next year as netapp supported.

                     

                    Regards,

                     

                    Keith

More Like This

  • Retrieving data ...

Legend

  • Correct Answers - 10 points
  • Helpful Answers - 5 points