Integration

Get Involved. Join the Conversation.

Topic

    B Carson
    Read File in Segments to REST Service insert to ATP database
    Topic posted November 15, 2019 by B Carson, tagged Database, Integration, Mapping, Orchestration, REST 
    17 Views, 6 Comments
    Title:
    Read File in Segments to REST Service insert to ATP database
    Summary:
    Read File in Segments to REST Service insert to database results in all rows written in each segment read
    Content:

    I have a very large file to import to my integration.  I am using FTP download and then Stage Read File in Segments to access the data that will be inserted into a database table in our Cloud ATP Oracle database.  The REST call is set up to utilize Batchload in order to post the 200 rows that each read in segment will pull.

    The result is that for each read, All rows in the inbound file are written to the database.  I believe the problem is in the way the elements are defined but cannot see that actual problem.  I am very new to OIC and am learning on the fly.  Any assistance will be appreciated.

    iar is attached.

    Version:
    19.3.3.190924.1600-31522
    Image:
    Document:

    Comment

     

    • Hemanth Lakkaraju

      I did not understand the problem. There is no for-each read. Do you mean for-each chunk? How many chunks are found when you tested? What's the number shown on chunk read activity when you tested the flow?

      • B Carson

        Correct.  I meant for-each chunk.  The specified number of rows for each chunk is 200, the total numbers of rows is 2004 so there were 11 chunks but I can't find in the activity stream where that might be reported.  I have attached a snippet of the log.

        Where could I see   the number shown on chunk read activity

         

        • Hemanth Lakkaraju

          It will be on UI when you open the instance in Monitoring --> Tracking.

          • B Carson

            I am looking at the Activity log under Monitoring --> Tracking.  I expanded the payload for my Stage File-Read request and response and have copied the contents to a text file.  Are you asking what the batchrow number is?  Or is there another item that I am not seeing within the log?  

            I have a suspicion that I have defined schemas incorrectly for download and stage read in segments functionality.

             

            • Hemanth Lakkaraju

              When you open the tracking instance, the flow design path will be marked green through which your actual message went through. On the stage chunk read activity, you will see the count of how many chunks it fetched.

              You can search in the attached text file - "ReadHRISWorkers emplid in chunk" that shows 11 times confirming 11 chunks are read.

              • B Carson

                Sorry to be so long in replying.  I was out of the office until today.

                Now I can see the count of chunks fetched in the flow design path.  Thank-you for enlightening me, this will be of use to me as I go along. 

                The count of 11 coincides with what I found in the output.  I will continue to dig into the mapping to see where it was that I pulled from that is causing my integration to pull too much data per chunk.