InfoSphere Information Server

Processing Web Services containing repeating group

responses with InfoSphere DataStage

2011 IBM Corporation

This presentation describes how to create an InfoSphere DataStage job to process

Web Services Responses with repeating groups. The example in this module is based on
DataStage 8.5 but the principles are the same for all versions of DataStage.

WebServicesRepeatingGroups.ppt

Page 1 of 32

Objectives

Understanding prerequisites
Import Web Services definitions
Identify repeating group within Web Service response
Build job to parse Web Service response

2011 IBM Corporation

The objectives of this presentation are to demonstrate the more advanced concepts and
techniques required for accessing published Web Services which use repeating group
structures. Note: the terminology 'Array' is also used to describe the group structure which
repeats.
The Web Service client stage does not have the capability to process repeating element
groups when used stand-alone. It must be paired with a separate XML processing stage to
parse the repeating element groups into multiple rows and columns.
This module will show how to identify web services that contain repeating group
structures, and how to create DataStage jobs that use both the Web Service Client and
XML Input stage to handle these structures.

WebServicesRepeatingGroups.ppt

Page 2 of 32

Understanding the prerequisites

Standards DataStage Web Services Stages support

SOAP 1.1 Binding over HTTP
Literal and SOAP-encoded web service arguments
RPC-style and document-style arguments
Web Service Metadata Import
Physical WSDL (Web Service Definition Language) files on disk
Direct HTTP-protocol (non-proxied) import of WSDL using URL

2011 IBM Corporation

This slide displays the web standards that are supported by the DataStage Web Services
stages, and the methods available for importing Web Service definitions. The steps in this
module will use Web Services published on the public internet. To complete these steps,
your local DataStage client AND your DataStage server will both need to have access to
the public internet.
This module covers more advanced topics. While it is not absolutely necessary, you
should first complete the Processing Web Services as a data source with InfoSphere
DataStage IBM Education Assistant module before you begin with this module.
The examples in this presentation are based on DataStage version 8.5, using Parallel
canvas jobs. You should be able to complete the module using other versions of
DataStage, or by using Server canvas jobs, however the screen captures and exact steps
may vary slightly.

WebServicesRepeatingGroups.ppt

Page 3 of 32

Import Web Services definitions (1 of 4)

Start Web Services Metadata Importer

2011 IBM Corporation

The first step is to import the Web Service Metadata. Start the Web Service Metadata
Importer by opening the DataStage Designer (or DataStage Manager in version 7). Next,
click Import from the main menu, then click Table Definitions, then click Web Services
WSDL Definitions.

WebServicesRepeatingGroups.ppt

Page 4 of 32

Import Web Services definitions (2 of 4)

Enter Web Service URL

Web service definition published to public internet:

http://wsf.cdyne.com/WeatherWS/Weather.asmx?wsdl

2011 IBM Corporation

The Web Service Metadata Importer should open as a new window. In this module, you
will use a web service that returns weather forecast information. The URL for the web
service is provided in the slide. To browse the Web Service, enter the URL for the Web
Service WSDL document into the Address field in the Web Service Metadata Importer and
press the Enter key.

WebServicesRepeatingGroups.ppt

Page 5 of 32

Import Web Services definitions (3 of 4)

Import GetCityForecastByZip

2011 IBM Corporation

Next, import the GetCityForecastByZIP operation. Click the [+] next to the entry for
WeatherSoap in the Web Services Explorer pane to expand. Locate the
GetCityForecastByZIP operation with the 'gears' icon next to it. Right-click this and select
'Import'. When the import finishes, press the Close button on the dialog box, and then the
Close button on the Web Services Metadata Importer window.

WebServicesRepeatingGroups.ppt

Page 6 of 32

Import Web Services definitions (4 of 4)

Verify import successful

2011 IBM Corporation

To verify that the import was successful, go to the repository view in the DataStage
Designer (or the DataStage Manager in version 7) and locate the Table Definitions folder.
Click to expand it. Then, locate the WebServices folder and expand it. You should have a
Weather folder there for the operation that was just imported. Expanding it should show all
of the entries labeled with GetCityForecastByZIP.

WebServicesRepeatingGroups.ppt

Page 7 of 32

Identify repeating group within Web Service response (1 of 6)

Manually examine WSDL Document
Examine imported definition
Column Names represent message element properties
Check value in Description Field
Shows message element name

2011 IBM Corporation

DataStage does not have a graphical tool for identifying repeating element groups within a
Web Service response. To do this, use a manual process of examining the WSDL (Web
Services Definition Language) document, assisted by the Web Service Metadata importer.
Any given WSDL document may contain multiple operations and many different complex
type definitions. To be sure you can identify what is needed, start by examining the
imported definition stored in the DataStage repository. Using the DataStage Designer (or
Manager with version 7) browse the repository under table definitions for the WebServices
folder. Expand Weather and locate the GetCityForecastByZIP_MSGOUT definition.
Double-click to open it and then select the Columns tab. The columns listed represent the
message element properties. The 'name' property identifies the message element name
that is returned from the web service in the Description field. As displayed on this slide, the
message element name is GetCityForecastByZIPSoapOut.

WebServicesRepeatingGroups.ppt

Page 8 of 32

Identify repeating group within Web Service response (2 of 6)

Load the WSDL document
Locate wdsl:message
Locate element GetCityForecastByZIPSoapOut
GetCityForecastByZipResponse wsdl:part element encapsulated in
GetCityForecastByZIPSoapOut

2011 IBM Corporation

From the DataStage Designer (or Manager in version 7), select Import from the main
menu, expand Table Definitions, and click Web Services WSDL Definitions. This should
open up the Web Service Metadata Importer as a new window. Enter the WSDL URL into
the Address field, the same as you did in the previous slide, and press the Enter' key to
load the document.
In the large main panel, scroll down to where the wsdl:message elements are being
defined and locate the element with name=GetCityForecastByZIPSoapOut which
matches the value you obtained on the previous slide. This element is highlighted in yellow
on this slide. Note that this message element encapsulates another wsdl:part element
which is circled in red, GetCityForecastByZipResponse.

WebServicesRepeatingGroups.ppt

Page 9 of 32

Identify repeating group within Web Service response (3 of 6)

Locate s:element with name GetCityForecastByZipResponse
Contains complexType tns:ForecastReturn
Occurs only once (maxOccurs=1)

10

2011 IBM Corporation

Scroll back up to where the wsdl:types are being defined and locate the s:element with
name=GetCityForecastByZIPResponse. This matches the wsdl:part from the previous
step.
This element contains a different complexType called tns:forecastReturn. maxOccurs=1
shows you that it occurs only once (its not repeating). You need to proceed further and
look at this type to see if it has any repeating groups.

WebServicesRepeatingGroups.ppt

Page 10 of 32

Identify repeating group within Web Service response (4 of 6)

Locate complexType with name=ForecastReturn
Multiple nonrepeating elements maxOccurs=1
Single complexType tns:ArrayOfForecast

11

2011 IBM Corporation

Continue scrolling through the types until you locate the complexType element with
name=ForecastReturn. This is highlighted in yellow on this slide. Once again, this
defines several elements which are not repeating as seen by maxOccurs=1. There is
also a single complexType tns:ArrayOfForecast which must be investigated as well.

WebServicesRepeatingGroups.ppt

Page 11 of 32

Identify repeating group within Web Service response (5 of 6)

Locate complexType with name=ArrayOfForecast
maxOccurs=unbounded shows repeating structure
Repeating group is type Forecast

12

2011 IBM Corporation

Next, locate the complexType with name=ArrayOfForecast. This type identifies a

repeating structure because the attribute maxOccurs=unbounded. This means it has no
upper limit. The repeating group is the type Forecast which can repeat any number of
times with no upper limit.

WebServicesRepeatingGroups.ppt

Page 12 of 32

Identify repeating group within Web Service response (6 of 6)

Locate the complexType with name=Forecast
Find element that will always be present
Use element with minOccurs=1

13

2011 IBM Corporation

Next, scroll one more time through the types and locate the complexType with
name=Forecast. This is the repeating structure found on the previous slide. The type is
highlighted in yellow on this slide.
When a DataStage job is built, a column in the XML Input Stage needs to be defined that
is mapped to an element within the XML document. The XML document will always be
present to trigger a new row to be created. Look for an element with minOccurs=1. For
this structure, the 'Date' element meets this requirement. It has minOccurs=1, which
means that it will always be present for each instance of this repeating group of elements.
You will see in the upcoming slides how this trigger is applied.
Before proceeding, click the close button in the lower right corner of the Web Service
Metadata Importer.

WebServicesRepeatingGroups.ppt

Page 13 of 32

Build job to parse Web Service response (1 of 17)

Create new DataStage parallel job

Source - Web Services Client
Xml input
Target Sequential file

14

2011 IBM Corporation

In the DataStage Designer, create a new parallel job. On the job canvas, add a web
services client as the source, followed by a XML Input stage and then a sequential file as
the target. The Web Services Client stage (WSClientPX) and the XML Input stage
(XMLInputPX) are located within the Real Time group in the Designer palette.

WebServicesRepeatingGroups.ppt

Page 14 of 32

Build job to parse Web Service response (2 of 17)

Define target file for sequential file stage

15

2011 IBM Corporation

Next, double click the sequential file stage. Click the Input tab at the top and then on the
Properties tab. Edit the path for the File property and provide the full path to the location
on your DataStage server where you want to write the file. This is a small file. After typing
the path, press the Enter key on the keyboard to submit. It is not necessary to change any
other options; the default values should be used. Press OK at the bottom of the dialog box
to save the changes.

WebServicesRepeatingGroups.ppt

Page 15 of 32

Build job to parse Web Service response (3 of 17)

Select GetCityForcastByZIP web service operation

16

2011 IBM Corporation

Open the Web Services Client stage. Select General tab. Click the 'Select Web Service
Operation' button. In the Web Service Browser window, select the 'Weather' web service
in the left pane, and the operations are listed in the right pane. Double click the
'GetCityForecastByZIP' operation in the right pane.

WebServicesRepeatingGroups.ppt

Page 16 of 32

Build job to parse Web Service response (4 of 17)

Verify Web services information populated

17

2011 IBM Corporation

Next, verify the Web Service information is populated. Click the Advanced button to view
the Web Service Information. If the information is not populated properly, go back and
verify the previous steps were completed correctly. Click OK when finished.

WebServicesRepeatingGroups.ppt

Page 17 of 32

Build job to parse Web Service response (5 of 17)

Load namespace and define input arguments
Enter valid US Zip Code

18

2011 IBM Corporation

Next, click the Output tab at the top and then on the Input Arguments tab. Press the Load
Arguments Information button and the grid for Namespace information should be
automatically populated. In the grid row for ZIP argument, enter a valid US zip code into
the Value column, such as 01460 for Littleton, Massachusetts.

WebServicesRepeatingGroups.ppt

Page 18 of 32

Build job to parse Web Service response (6 of 17)

Manually load column definition to hold entire SOAP message body

19

2011 IBM Corporation

With the Output tab still selected at the top, select the Columns tab. Press the Load button
and a Load Columns message box will appear, confirming if you want to proceed with
manual load. Click Yes.

WebServicesRepeatingGroups.ppt

Page 19 of 32

Build job to parse Web Service response (7 of 17)

Locate SOAPbody definition

20

2011 IBM Corporation

Browse the Table definitions and expand the Built-in folder and then the Examples folder.
Double click the SOAPbody definition to select it.

WebServicesRepeatingGroups.ppt

Page 20 of 32

Build job to parse Web Service response (8 of 17)

Manually load column definition to hold entire SOAP message body

21

2011 IBM Corporation

On the Select Columns dialog, SOAPbody should appear in both panels. Press the Ok
button to add the column.

WebServicesRepeatingGroups.ppt

Page 21 of 32

Build job to parse Web Service response (9 of 17)

22

2011 IBM Corporation

The columns grid should now be populated with a single column, SOAPBody. This column
will hold the entire XML contents of the SOAP Body response from the web service.

WebServicesRepeatingGroups.ppt

Page 22 of 32

Build job to parse Web Service response (10 of 17)

Configure user-defined output message to output column

23

2011 IBM Corporation

With the Output tab still selected at the top, select the Output Message tab. DO NOT load
the namespace information. Instead, click the box for User-Defined Message. In the drop
down menu, select the SOAPBody column you configured in the previous slide. Close the
Web Service Stage by pressing the OK button.

WebServicesRepeatingGroups.ppt

Page 23 of 32

Build job to parse Web Service response (11 of 17)

Configure transformation settings for XML Input stage
Repeating Elements Requires
Include namespace declaration

24

2011 IBM Corporation

Next, open the XML Input stage. With the Stage tab selected at the top, select the
Transformation Settings tab. As displayed on this slide, check the boxes for Repetition
Element Required and Include Namespace declaration. Ensure the rest of the boxes are
unchecked.
Click the Load button to import the namespace information. Browse the repository tree
under Table definitions and then the WebServices folder. Expand Weather, and locate the
GetCityForecastByZIP_OUT definition. Double click to select it.

WebServicesRepeatingGroups.ppt

Page 24 of 32

Build job to parse Web Service response (12 of 17)

Configure XML Input stage source
Select XML Source column SOAPbody
Select XML Document

25

2011 IBM Corporation

Select the Input tab at the top, then select the XML Source tab. In the drop down menu for
XML Source column, select SOAPBody. For the Column content option, choose XML
document.

WebServicesRepeatingGroups.ppt

Page 25 of 32

Build job to parse Web Service response (13 of 17)

Configure output transformation settings
Inherit stage properties

26

2011 IBM Corporation

Select the Output tab at the top, then select the Transformation Settings tab. Check the
box to Inherit stage properties. The remaining options should now appear disabled as
displayed in the screen capture on this slide.

WebServicesRepeatingGroups.ppt

Page 26 of 32

Build job to parse Web Service response (14 of 17)

Load output columns

27

2011 IBM Corporation

With the Output tab selected at the top, select the Columns tab. Press the Load button to
import columns. Browsing the repository, expand the Table definitions and WebServices
folders, then the Weather folder. Double-click the GetCityForecastByZIP_OUT definition to
import it.

WebServicesRepeatingGroups.ppt

Page 27 of 32

Build job to parse Web Service response (15 of 17)

Load output columns

28

2011 IBM Corporation

In the Select Columns dialog all of the columns are listed on both the left and right side.
Click the OK button to load all of the columns.

WebServicesRepeatingGroups.ppt

Page 28 of 32

Build job to parse Web Service response (16 of 17)

Configure Date as repeating element
Set Key = Yes

29

2011 IBM Corporation

The grid should now be populated with all the columns from the previous step.

On slide 13, you identified a repeating element, date, which will always be present. You

determined this from examining the WSDL document. Locate the column named 'Date'

within the grid. For the Key property, choose 'Yes' from the drop down menu. This step

configures Date as the repeating element.

WebServicesRepeatingGroups.ppt

Page 29 of 32

Build job to parse Web Service response (17 of 17)

Save, compile, and run job

30

2011 IBM Corporation

Save and compile the job. To enable performance statistics, right click anywhere on the
blank area of the job canvas and select Show Performance Statistics. Run the job from
Designer. The link should turn green and report one row processed as input to the XML
stage and multiple rows written to the sequential file stage. Right click the Sequential file
stage and view data to see the individual column values returned.
Note: You may find that the view data browser displays only one column of data at a time.
This is because the maximum lengths for the fields are set to 255. To see more than one
column at a time, use the mouse to resize the column widths in the view data browser.

WebServicesRepeatingGroups.ppt

Page 30 of 32

Feedback

Your feedback is valuable

You can help improve the quality of IBM Education Assistant content to better meet your
needs by providing feedback.
Did you find this module useful?
Did it help you solve a problem or answer a question?
Do you have suggestions for improvements?

Click to send email feedback:

mailto:[email protected]?subject=Feedback_about_WebServicesRepeatingGroups.ppt

This module is also available in PDF format at: ../WebServicesRepeatingGroups.pdf

31

InfoSphere Information Server

2011 IBM Corporation

You can help improve the quality of IBM Education Assistant content by providing
feedback.

WebServicesRepeatingGroups.ppt

Page 31 of 32

Trademarks, disclaimer, and copyright information

IBM, the IBM logo, ibm.com, DataStage, and InfoSphere are trademarks or registered trademarks of International Business Machines Corp., registered
in many jurisdictions worldwide. Other product and service names might be trademarks of IBM or other companies. A current list of other IBM
trademarks is available on the web at "Copyright and trademark information" at http://www.ibm.com/legal/copytrade.shtml
Other company, product, or service names may be trademarks or service marks of others.
THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE
MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED
"AS IS" WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON IBMS CURRENT
PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY IBM WITHOUT NOTICE. IBM SHALL NOT BE RESPONSIBLE FOR
ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION.
NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, NOR SHALL HAVE THE EFFECT OF, CREATING ANY WARRANTIES OR
REPRESENTATIONS FROM IBM (OR ITS SUPPLIERS OR LICENSORS), OR ALTERING THE TERMS AND CONDITIONS OF ANY AGREEMENT
OR LICENSE GOVERNING THE USE OF IBM PRODUCTS OR SOFTWARE.
Copyright International Business Machines Corporation 2011. All rights reserved.

32

WebServicesRepeatingGroups.ppt

2011 IBM Corporation

Page 32 of 32