Integration

Get Involved. Join the Conversation.

Topic

    Thirunavukarasur
    OIC on-prim Agent performance improvement
    Topic posted June 26, 2018 by Thirunavukarasur Bronze Medal: 1,250+ Points, tagged Agent 
    256 Views, 12 Comments
    Title:
    OIC on-prim Agent performance improvement
    Summary:
    How to improve the performance of OIC on-prim agent
    Content:

    We have couple of integrations which times out intermittently and , any suggestion on  how to improve the performance of on-prim agent as well as OIC.  Thank you so much for the help.

    Thanks

    Ravi

    Comment

     

    • Ankur Jain

      Hi Ravi,

      Can you explain little bit more which type of integration you have which are causing time out issue. Also can you share logs.

      Regards,

      Ankur

      https://www.techsupper.com

      • Thirunavukarasur

        Ankur,

        This is what we see in OIC managed server log on the cloud.  Agent monitoring page shows status green.  checked the on-prim agent log.  Looks good.

        [2018-06-27T12:53:55.004+00:00] [VMCEBSJA_server_1] [ERROR] [OSB-381990] [oracle.osb.transports.jca.jcatransport] [tid: [ACTIVE].ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: jaggaer_test] [ecid: 649a8c62-def7-40c7-adb6-c7dd68ada531-00167ce0,0:13] [partition-name: DOMAIN] [tenant-name: GLOBAL] [oracle.soa.tracking.FlowId: 16400268] [oracle.soa.tracking.InstanceId: 1441002] [oracle.soa.tracking.SCAEntityId: 280071] [composite_name: VMCPURCHASEORDERIMPORT!01.00.0000] [FlowId: 0000MH1Ikex3b615VvXBic1RAyuc000jpt] Invoke JCA outbound service failed with application error, exception: No response received within response time out window of 240 seconds. Agent may not be running, or temporarily facing connectivity issues to Oracle Integration Cloud Service. Please check the health of the Agent in Agent Monitoring page.

        Thanks

        Ravi

      • Thirunavukarasur

        Ankur,

        here is the on-prim agent log.  Eventhough we see the payload information in the agent log.  Error message we see is "Request Payload is null".  Wondering what can cause this.

         

        Method: log] JCABinding <outbound> Connection Properties :
        [cloudAdapterName : database]
        [MaxConnections : 35]
        [ConnectionString : 2B731F769540C4FE1208361ADF9496D4336FD51E443659802714EECCA09A4CB01DE5BEA63D25B401AC9204A83513C9E3ED0AD589412BDC1FBA48B372BF76C3DC]
        [MaxReadConnections : 15]
        [PlatformClassName : org.eclipse.persistence.platform.database.Oracle10Platform]
        [csfkey : 6923c9ce4ef67c47ce04a37b15661302c3f7804f2321e1627e6bac11b0903c37]
        [DriverClassName : oracle.jdbc.OracleDriver]
        [MinConnections : 5]
        [csfMap : oracle.cloud.adapter]
        [MinReadConnections : 5]

        [2018-06-27T12:54:24.068Z] [FINE] [ThreadID: 19] [Logger: oracle.soa.adapter] [SRC Class: oracle.tip.adapter.fw.log.LogManagerImpl; Method: log] JCABinding <outbound> Message HeadersMap is null
        [2018-06-27T12:54:24.069Z] [FINE] [ThreadID: 19] [Logger: oracle.soa.adapter] [SRC Class: oracle.tip.adapter.fw.log.LogManagerImpl; Method: log] JCABinding <outbound> Integration Payload :
         <InputParameters xmlns="http://xmlns.oracle.com/cloud/adapter/database/INSERTPO_REQUEST/types" xmlns:flt="http://xmlns.oracle.com/cloud/generic/service/fault" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:tns="http://xmlns.oracle.com/cloud/adapter/database/INSERTPO_REQUEST" xmlns:plt="http://schemas.xmlsoap.org/ws/2003/05/partner-link/" xmlns:nsmpr0="http://xmlns.oracle.com/cloud/adapter/nxsd/surrogate/request/GETPOS/" xmlns:nstrgmpr="http://xmlns.oracle.com/cloud/adapter/database/INSERTPO_REQUEST/types">

         

        [2018-06-27T12:54:24.068Z] [FINE] [ThreadID: 19] [Logger: oracle.soa.adapter] [SRC Class: oracle.tip.adapter.fw.log.LogManagerImpl; Method: log] JCABinding <outbound> Message HeadersMap is null
        [2018-06-27T12:54:24.069Z] [FINE] [ThreadID: 19] [Logger: oracle.soa.adapter] [SRC Class: oracle.tip.adapter.fw.log.LogManagerImpl; Method: log] JCABinding <outbound> Integration Payload :
         <InputParameters xmlns="http://xmlns.oracle.com/cloud/adapter/database/INSERTPO_REQUEST/types" xmlns:flt="http://xmlns.oracle.com/cloud/generic/service/fault" xmlns:wsdl="http://schemas.xmlsoap.org/wsdl/" xmlns:tns="http://xmlns.oracle.com/cloud/adapter/database/INSERTPO_REQUEST" xmlns:plt="http://schemas.xmlsoap.org/ws/2003/05/partner-link/" xmlns:nsmpr0="http://xmlns.oracle.com/cloud/adapter/nxsd/surrogate/request/GETPOS/" xmlns:nstrgmpr="http://xmlns.oracle.com/cloud/adapter/database/INSERTPO_REQUEST/types">

         


        [2018-06-27T12:54:24.069Z] [FINE] [ThreadID: 19] [Logger: oracle.soa.adapter] [SRC Class: oracle.tip.adapter.fw.log.LogManagerImpl; Method: log] JCABinding <outbound> Request payload is null

    • Thirunavukarasur

      Ankur,

      This is what we see in OIC managed server log on the cloud.  Agent monitoring page shows status green.  checked the on-prim agent log.  Looks good.

      [2018-06-27T12:53:55.004+00:00] [VMCEBSJA_server_1] [ERROR] [OSB-381990] [oracle.osb.transports.jca.jcatransport] [tid: [ACTIVE].ExecuteThread: '11' for queue: 'weblogic.kernel.Default (self-tuning)'] [userId: jaggaer_test] [ecid: 649a8c62-def7-40c7-adb6-c7dd68ada531-00167ce0,0:13] [partition-name: DOMAIN] [tenant-name: GLOBAL] [oracle.soa.tracking.FlowId: 16400268] [oracle.soa.tracking.InstanceId: 1441002] [oracle.soa.tracking.SCAEntityId: 280071] [composite_name: VMCPURCHASEORDERIMPORT!01.00.0000] [FlowId: 0000MH1Ikex3b615VvXBic1RAyuc000jpt] Invoke JCA outbound service failed with application error, exception: No response received within response time out window of 240 seconds. Agent may not be running, or temporarily facing connectivity issues to Oracle Integration Cloud Service. Please check the health of the Agent in Agent Monitoring page.

      Thanks

      Ravi

      • Ankur Jain

        Hi Ravi,

        This error usually occurs when you are sending request through agent to DB /  any service etc and agent is not getting response back in 240 seconds from DB / service.

        From the logs it looks like you are performing some DB operation which is taking more than 4 minutes. 

         

        Please try below 

        1) Execute the query directly to sql developer and see the response time. If it is less than 4 minutes which means query is ok else tune the query 

        2) Try restarting the agent and see if it resolves the issue.

        Oracle recommendation is agent should be in the same network as on premises application to reduce network latency.

        Regards,

        Ankur 

      • Nagireddy Tadi

        When monitoring page shows status green, possible reason for this time out error is,  runtime request (db query/procedure) taking more  time.  When you see this exception on OIC can you check agent logs at the same time frame and see whether runtime request is getting completed  with in 240 seconds or not.

      • Amandeep Mahajan

        Hi Ravi,

        Adding to what Nagi mentioned below;

        1) Are the intermittent failures always for the following flow VMCPURCHASEORDERIMPORT?

        If yes, we can isolate it to that specific flow on the on-premises agent as well and see if any interaction is taking longer than the timeout window which is 240s. 

        NOTE: I see you have enabled TRACE for all loggers, your concern related to null payload can be ignored as it is just getting printed out FINER log level.  

        Thanks

         

         

        • Thirunavukarasur

          Hi Amandeep,

          The failure is not consistent.  we are experiencing failures across integrations.  Any idea where this timeout is defined.

           

          Thanks

          Ravi

          • Amandeep Mahajan

            If this is across integrations, there may be something else that need to be looked at as well. IMO, timeout is not an issue, 4 mins is a long time for your endpoint invoke to finish but if it is taking longer than that you may need to root cause as to why it is taking that much longer.

            Do you know what your aveage DB invoke times are?

            • Thirunavukarasur

              Amandeep,

              I am sorry, we are new to OIC.  where can we find this information.

               

              thanks

              • Amandeep Mahajan

                My question was specific to your backend DB. Have you looked at the AWR? Does it point to any bottleneck issue at the DB end?

                In addition, you can share your agent logs (for the period when timeouts occur) with us and we can review the same...

    • Thirunavukarasur

      Hello All,

      Want to share what we have found. we are on a two node RAC.   We enabled  the logon trigger for the interfacing account in eBS database.  Did a tkprof and found the below query is causing the timeout. 

      SELECT /*+ NOPARALLEL */ 1, U.NAME, O.NAME, O.OID$
      FROM
       SYS. "_CURRENT_EDITION_OBJ" O, SYS.USER$ U, SYS.TYPE$ T WHERE T.SUPERTOID =
        :B1 AND T.TVOID = O.OID$ AND O.SUBNAME IS NULL AND O.OWNER# = U.USER#


      call     count       cpu    elapsed       disk      query    current        rows
      ------- ------  -------- ---------- ---------- ---------- ----------  ----------
      Parse       14      0.02       0.66          0          0          0           0
      Execute     14      0.00       0.00          0          0          0           0
      Fetch       14      9.17     266.60     500976     501816          0           0
      ------- ------  -------- ---------- ---------- ---------- ----------  ----------
      total       42      9.19     267.26     500976     501816          0           0

       

      After analysing further , noticed the init parameter "parallel_force_local " was set to false.  We changed "parallel_force_local " to True.  Looks like that fixed our timeout issue. 

      Thanks

      Ravi