IBM Software Group

WebSphere DataPower SOA Appliances

and XSLT (Part 1 of 2) - Tools
Hermann Stamm-Wilbrandt ([email protected])
DataPower XML Compiler Developer, L3
6 July 2010

WebSphere Support Technical Exchange

IBM Software Group

Agenda
DataPower tools

built-in
others
Appendix

WebSphere Support Technical Exchange

2 of 23

IBM Software Group

Bookmarking DataPower screens for easy access

Multibox bookmarklets they avoid navigation through left navbar in WebGUI:
Stylesheet Cache (3.7.x bookmarklet / 3.8.0 bookmark / 3.8.0 bookmarklet)
javascript:window.mainFrame.genericStatusRequest('StylesheetCachingSummary',%
20'genericStatus')
https://yourbox:webguiport/status/StylesheetCachingSummary
javascript:void(location=window.location.href.replace(/(^[^:]*:\/\/[^/]*).*$/g,'$1')
+'/status/StylesheetCachingSummary')
Xpath Tool (3.7.x bookmark / 3.8.0 bookmark / 3.8.0 bookmarklet)
.../TaskTemplates/SelectXPath.xml?...
https://boxip:webguiport/service/SelectXPath?
popup=true&newObjPopup=true&newObjPopupInput=input_XPath&step=getstarted&popuplocation=
javascript:void(window.open(window.location.href.replace(/(^[^:]*:\/\/[^/]*).*$/g,'$1')
+'/service/SelectXPath?
popup=true&newObjPopup=true&newObjPopupInput=input_XPath&step=getstarted&popuplocation=').focus())
Works for FF, IE, Chrome and Safari browsers
Source: Bookmarklets and WebGUI posting on developerWorks DataPow er Forum

WebSphere Support Technical Exchange

3 of 23

IBM Software Group

Stylesheet profiling (1/2)

Configure profiling of a stylesheet by
a compile option policy (for its XML Manager)
and a matching Profile Rule there.

Sending requests against the service containing

the stylesheet gives profiling information.

This may be looked up in WebGUI under

StatusXML ProcessingStylesheet Profiles

Flushing the XML Manager resets the profile

information.

<dp:profile> allows to define profiling regions

Source: Appendix D. Compile Options Policy configuration/Profiling overview (CommandReference)

WebSphere Support Technical Exchange

4 of 23

IBM Software Group

Stylesheet profiling (2/2)

Sample of improvement (replace pure XSLT implementation of hex-to-text by an

implementation making use of extension functions -- template hex-to-text gets inlined)

WebSphere Support Technical Exchange

5 of 23

IBM Software Group

Stylesheet tracing
Configure tracing of a stylesheet by
a compile option policy (for its XML Manager)
and a matching Debug Rule there.

Sending a request against the service containing

the stylesheet returns a trace HTML page.

If needing a (less informative) trace facility not

affecting MultiStep processing you may want to
have a look at trace.xslt by Oliver Becker.

Source: CommandReference / trace.xslt by Oliver Becker

WebSphere Support Technical Exchange

6 of 23

IBM Software Group

DataPower XML data processing (1/4)

Document Object Model (DOM)
Reads all of the XML data and forms the corresponding tree in memory.
After building the tree, the application program processes the data.
Processing consists of one or more traversals of the data in arbitrary order.
As it performs these traversals, the application program generates the output.

Simple API for XML model (SAX)

Combines the reading of the data with the processing.
Reading the nodes the application processes them and then deletes its storage.
In other words, the tree is not built. Instead, the data is processed on-the-fly.
All schema validation in the DataPower appliance uses the SAX model.

Streaming model
Hybrid technique, the application starts processing in SAX model manner.
If needed the application program builds trees for that part of the data as the DOM.
After processing of this tree completes it is immediately removed.
Flexibility of the DOM model with some of the storage advantages of the SAX model.
Source: OptimizingThroughStreaming-v1.pdf

WebSphere Support Technical Exchange

7 of 23

IBM Software Group

Streaming (2/4)
This stylesheet is streamable.
The commenting out was
necessary (double use of .).

You can send documents

containing millions of records
through the box without noticeable
memory increase.

Source: Find stylesheet and script for generating large input data in appendix

WebSphere Support Technical Exchange

8 of 23

IBM Software Group

Partial Streaming (3/4)

This template is not streamable SAX like.
Reason is the <xsl:choose> statement.
But the stylesheet is partially streamable.
A single record is processed DOM like.
After processing the memory is given
back immediately.

Again you can send documents

containing millions of records through the
box without noticeable memory increase

Source: Find stylesheet and script for generating large input data in appendix

WebSphere Support Technical Exchange

9 of 23

IBM Software Group

Determine why a stylesheet is not streamable (4/4)

A stylesheet will only be processed streaming if
a compile option policy is configured for its XML Manager,
a matching Streaming Rule or Attempt Streaming Rule is configured there
and the stylesheet itself is streamable.

In order to determine whether a stylesheet is streamable

use matching Streaming Rule (to enforce streaming mode)
send any traffic against a service containing an action using the stylesheet
in case the stylesheet is not streamable you will get a generic error in the log.

For stylesheets not being streamable find the reason by

switch to StatusXML ProcessingStylesheet Status in WebGUI
inspect stylesheet's entry
lookup column Compiler Log for detailed errors

WebSphere Support Technical Exchange

10 of 23

IBM Software Group

XPath Tool
In WebGUI's left navbar

click on "Objects->XML
Processing->XPath
Routing Map"
Click on "Add" and select
the "Rules" tab
Click "Add" again, a popup
opens
Click "XPath Tool" in the
popup, "Xpath Tool" popup
opens.

Or use bookmark
Upload XML file
Click somewhere (eg. Log)
Inspect generated Xpath
and take it as a start

WebSphere Support Technical Exchange

11 of 23

IBM Software Group

Agenda
DataPower tools

built-in
others
Appendix

WebSphere Support Technical Exchange

12 of 23

IBM Software Group

Using xpath tool to gain information on XML and XSL files

$ xpath "string(//*[name()='xsl:copy-of']/@select)" xpath.xsl
dyn:evaluate(string((//comment())[last()]))
$
get insight in stylesheet

$ xpath "count(//*[name()='xsl:for-each']//*[name()='xsl:for-each']//*[name()='xsl:foreach']//*[name()='xsl:for-each']//*[name()='xsl:for-each']//*[name()='xsl:for
each']//*[name()='xsl:for-each']//*[name()='xsl:for-each']//*[name()='xsl:foreach']//*[name()='xsl:for-each']//*[name()='xsl:for-each']//*[name()='xsl:for-each'])"
HugeXslFor-each.xsl
4
$
uncovering (structural) performance issues in stylesheets

xpath "/*/*/*[name()='dp:response']/*[name()='dp:status']/StylesheetStatus/Log[text()]"
returns all warnings/errors retrieved by get-status SOMA call for compiled stylesheets
Source: xpath tool posting on developerWorks DataPow er Forum

WebSphere Support Technical Exchange

13 of 23

IBM Software Group

Scripting DataPower
DataPower Scripting is preferably done over the XML Management Interface
This is a really good RedPaper on this topic:
WebSphere DataPower SOA Appliance: The XML Management Interface
Rolf Wittich
http://www.redbooks.ibm.com/abstracts/redp4446.html

Internal CLI scripting is possible, too:

store a CLI script eg. as local:///script.cfg
open a CLI session to the box
do exec local:///script.cfg

External CLI scripting can be done eg. with netcat command (nc, Linux/Cygwin)
have Telnet service enabled (on intranet interface)
store sequence of entries of a CLI session into eg. cli.txt
execute nc yourbox telnetport < cli.txt

Executing CLI commands from within a stylesheet is possible (posting link below)
While DataPower currently has no raw TCP support on the front side it HAS
support for raw TCP on the back side, the protocol "tcp:" ( Telnet service)
Source: Executing CLI commands f rom w ithin a stylesheet posting on developerWorks DataPow er Forum

WebSphere Support Technical Exchange

14 of 23

IBM Software Group

Processing UTF-8 URL-encoded URIs with WebSphere DataPower

DataPower action "convert-http"
creates XML document for HTTP form data
converts non-XML query parameters to XML
assumption is ISO-8859-1 encoding

Workaround for UTF-8, see technote referenced at bottom

Main aspects of workaround
Accessing the URL
select="dp:variable('var://service/URI')" /> ...
Tokenizing the query
<xsl:for-each select="str:tokenize(substring-after($url,'?'),'&amp;')">
Decoding the UTF-8 URL-encoded data
select="str:decode-uri(substring-after(.,'='))"

DataPower provides lots of Extension functions/elements, you may want to have a look into
the Extension Functions Catalog ...
Source: Processing UTF-8 URL-encoded URIs w ith WebSphere DataPow er technote

WebSphere Support Technical Exchange

15 of 23

IBM Software Group

Enhancing near SOAP With Attachment data

SWA

(Soap With Attachments)

MIME boundaries
content type "multipart/related"

sa

Content-ID for each/most sections

arbitrary content including binary
first section SOAP

Strip
Strip
Attachments
Attachmen
ts

pl e

ap

pli

ca

tio
n

HTTP form data (HTTP POST)

content type "multipart/form-data"
Content-Disposition for sections

swaform tool
Non-XML request type policy
replaces "Content-Type" multipart/form-data by multipart/related
replaces 'Content-Disposition ... name="xyz" ...' by 'Content-Id: <xyz>'
prepends root section with "dummy" SOAP message
Source: sw aform -- using HTTP f orms as generator f or "Soap With Attachments" posting on developerWorks DataPow er Forum

WebSphere Support Technical Exchange

16 of 23

IBM Software Group

Processing embedded stylesheets

Not many customers are using

Transform (Using Processing Instruction) at all.

DataPower does not support processing of embedded stylesheets out of the box.

Use an "Extract Using XPath" action to extract the stylesheet into a context "ctx".
The XPATH to use is
//*[attribute::*[local-name()='id']=substring-before(substring-after(/processing

instruction()[local-name()='xml-stylesheet'],'href="#'),'"')]
Be careful with the single and double quotes.
It extracts the stylesheet by the id referenced by href in xml-stylesheet processing
instruction into context "ctx".
Next use a xform action reading from INPUT with processing control file "ctx".
It is not possible to create this in WebGUI directly.
After having selected any xform action (eg. store://identity.xsl) you have to
edit the action in CLI and execute "transform ctx" to make the xform action take
the stylesheet from context "ctx"
or via WebGUI using Objects menu, select ObjectsXML ProcessingProcessing
Action, the Transform is a text field in this page and you input ctx there.

Source: 2.7 Embedding Stylesheets (XSLT 1.0 spec) / cdcatalog.xml

WebSphere Support Technical Exchange

17 of 23

IBM Software Group

Agenda
DataPower tools

built-in
others
Appendix

WebSphere Support Technical Exchange

18 of 23

IBM Software Group

Appendix (for copy&paste)

dbgen.pl
#!/usr/bin/perl
$size = shift;
if ($size eq "")
{
die "usage: dbgen.pl [size]\n";
}
@firstnames = ("Al", "Bob", "Charles", "David",
"Egon", "Farbood",
"George", "Hank", "Inki", "James");
@lastnames = ("Aranow", "Barker", "Corsetti",
"Dershowitz", "Engleman",
"Franklin", "Grice", "Haverford",
"Ilvedson", "Jones");
@states = ("AL", "AK", "AZ", "AR", "CA", "CO", "CT",
"DE", "FL", "GA",
"HI", "ID", "IL", "IN", "IA", "KS", "KY",
"LA", "ME", "MD",
"MA", "MI", "MN", "MS", "MO", "MT", "NE",
"NV", "NH", "NJ",
"NM", "NY", "NC", "ND", "OH", "OK", "OR",
"PA", "RI", "SC",
"SD", "TN", "TX", "UT", "VT", "VA", "WA",
"WV", "WI", "WY");
print "<?xml version=\"1.0\"?>\n";
print "\n";
print "<table>\n";
for ($i=0; $i<$size; $i++)
{
$first = $firstnames [$i % 10];
$last = $lastnames [($i / 10) % 10];
$state = $states [($i / 100) % 50];
$zip = 22000 + $i / 5000;
printf
printf
printf
printf
printf
100) + 1;
printf
printf
printf
printf
}

dbsql.xsl
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/
Transform"
>
<xsl:template match="table">
<document>
<xsl:apply-templates select="row"/>
</document>
</xsl:template>
<xsl:template match="row">
<xsl:apply-templates select="id"/>
<xsl:apply-templates select="firstname"/>
<xsl:apply-templates select="lastname"/>
<xsl:apply-templates select="street"/>
<xsl:apply-templates select="city"/>
<xsl:apply-templates select="state"/>
<xsl:apply-templates select="zip"/>
<xsl:text>&#x0A;</xsl:text>
</xsl:template>

"
"
"
"
"

<row>\n";
<id>%04d</id>\n", $i;
<firstname>$first</firstname>\n", $i;
<lastname>$last</lastname>\n", $i;
<street>%d Any St.</street>\n", ($i %

<xsl:template match="id|firstname|lastname|
street|city|state|zip">
<xsl:value-of select="."/>
<xsl:text>&#x0A;</xsl:text>
</xsl:template>

"
"
"
"

<city>Anytown</city>\n";
<state>$state</state>\n";
<zip>%d</zip>\n", $zip;
</row>\n";

</xsl:stylesheet>

print "</table>\n";

WebSphere Support Technical Exchange

dbsql1.pl
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
>
<xsl:template match="table">
<document>
<xsl:apply-templates select="row"/>
</document>
</xsl:template>
<xsl:template match="row">
<xsl:choose>
<xsl:when test="contains('CA,NY,WY',state)">
<xsl:apply-templates select="id"/>
<xsl:apply-templates select="firstname"/>
<xsl:apply-templates select="lastname"/>
<xsl:apply-templates select="state"/>
</xsl:when>
<xsl:otherwise>
<xsl:apply-templates select="id"/>
<xsl:apply-templates select="firstname"/>
<xsl:apply-templates select="lastname"/>
<xsl:apply-templates select="street"/>
<xsl:apply-templates select="city"/>
<xsl:apply-templates select="state"/>
<xsl:apply-templates select="zip"/>
</xsl:otherwise>
</xsl:choose>
<xsl:text>&#x0A;</xsl:text>
</xsl:template>
<xsl:template match="id|firstname|lastname|street|
city|state|zip">
<xsl:value-of select="."/>
<xsl:text>&#x0A;</xsl:text>
</xsl:template>
</xsl:stylesheet>

19 of 23

IBM Software Group

Summary
Built-in DataPower tools (Stylesheet profiling and
tracing, XPath tool)
DataPower XML Data Processing (streaming,
partial streaming)
Other DataPower tools (xpath tool, scripting,
processing UTF-8 URL-encoded URIs, Enhancing
near SOAP With Attachment data, processing
embedded stylesheets)

Next webcast (2/2) will be on Tips and Tricks

WebSphere Support Technical Exchange

20 of 23

IBM Software Group

Additional WebSphere Product Resources

Learn about upcoming WebSphere Support Technical Exchange webcasts, and access
previously recorded presentations at:
http://www.ibm.com/software/websphere/support/supp_tech.html

Discover the latest trends in WebSphere Technology and implementation, participate in

technically-focused briefings, webcasts and podcasts at:
http://www.ibm.com/developerworks/websphere/community/

Join the Global WebSphere User Group Community:

http://www.websphere.org

Access key product show-me demos and tutorials by visiting IBM Education Assistant:
http://www.ibm.com/software/info/education/assistant

View a webcast replay with step-by-step instructions for using the Service Request (SR)
tool for submitting problems electronically:
http://www.ibm.com/software/websphere/support/d2w.html

Sign up to receive weekly technical My Notifications emails:

http://www.ibm.com/software/support/einfo.html

WebSphere Support Technical Exchange

21 of 23

IBM Software Group

We Want to Hear From You!

Tell us about what you want to learn
Suggestions for future topics
Improvements and comments about our webcasts
We want to hear everything you have to say!
Please send your suggestions and comments to:
[email protected]

WebSphere Support Technical Exchange

22 of 23

IBM Software Group

Questions and Answers

WebSphere Support Technical Exchange

23 of 23