Pebble Flow Workflow Server

The Pebble Flow Server is a private cloud deployable server that can expose pebbles as services via RESTful APIs.

Deploying Pebble Flow on Microsoft's Azure Cloud

Create the resource

The Pebble Flow Workflow Server is deployable as an Azure resource. It is available on the Azure Marketplace. Click on "Create a resource"

Azure services's "Create a Resource" button

Azure services's "Create a Resource" button


Search for the Pebble Flow Workflow Server

Click on the Pebble Flow Workflow Server card.

Search for "Pebble Stream" to find the Pebble Flow server on Azure Marketplace.

Search for "Pebble Stream" to find the Pebble Flow server on Azure Marketplace.

Begin Pebble Flow deployment

Point-and-click deployment via the Azure Marketplace UI is straightforward and completed in three steps. Choose your appropriate plan, then click the blue "Create" button to start configuring the Pebble Flow deployment.

Click on the 'Create' button to begin Azure deployment.

Click on the 'Create' button to begin Azure deployment.

👍

Choose the plan that best fits your organization's needs.

Step 1 - select your deployment subscription and choose resource names

In the first step, select your subscription and a unique resource group for deployment. You should isolate Pebble Flow's deployment from your other Azure resource groups. Choose your App Service plan and select a storage account type that meets your business continuity need.

$186.15/

Choose your subscription, then select names for the deployable resource group.

Choose your subscription, then select names for the deployable resource group.

👍

Be sure to choose plans that support autoscaling if you want to use that feature.

Step 2 - Enter the API key and specify the admin user

Enter your API key (contact us to get this), along with the email address and name of the Pebble Flow administrator. The deployment will automatically create an account for this user. Make sure to enter the correct email address at this step.

Please enter the API key, a valid admin user email and name, and the organization's name.

Please enter the API key, a valid admin user email and name, and the organization's name.

Step 3 - optionally copy metadata from a previous installation.

If there was a previous Pebble Flow installation, you may copy its pebble resources to this new installation. If you don't want to do this, uncheck "Copy data from existing storage account" and proceed to the next step.

 Copy users, compiled pebbles, spreadsheets, workflows, authorizations, and API endpoints to the new installation.

Copy users, compiled pebbles, spreadsheets, workflows, authorizations, and API endpoints to the new installation.

❗️

Request, job, and scenario results from the previous install are not copied.

Review configuration, accept terms, then deploy

On the final screen, you must agree to the terms and conditions for using Pebble Flow on the Azure Marketplace.

Agree to the terms and conditions for running resources procured from Azure's Marketplace.

Agree to the terms and conditions for running resources procured from Azure's Marketplace.

$186.15/The Pebble Flow Workflow Server deployment should be completed in 30 minutes or less.

Autoscaling Pebble Flow on Microsoft's Azure Cloud

Pebble Flow is designed to run as a dynamically distributed service on Azure. Azure App Service's built-in autoscaling can be used to manage costs. Pebble Flow's API uses Azure Blobstore's queues to manage request traffic, and Azure App Service is designed to monitor queue traffic for scaling cues.

Configuring Azure App Service autoscaling via the portal is simple.

You can use Azure's portal to configure scaling rules based on pending Pebble Flow request message counts.

You can use Azure's portal to configure scaling rules based on pending Pebble Flow request message counts.

One way to configure your scaling capability using Azure's portal is to scale up the number of Pebble Flow servers as long as the average number of pending requests is above a certain threshold. Scaling down could be triggered by completely draining the request queue. Of course, the maximum number of servers can be capped, regardless of the number of pending requests.

👍

Make sure to choose an App Service sku that supports autoscaling. If you don't, you can always change your plan.

The Pebble Flow API

Invoking a Pebble via the Pebble Flow Service

Whenever a pebble is compiled, the Pebble Flow server exposes the pebble to the enterprise via a regular HTTPS interface. The Pebble Flow API is hypermedia-driven; each call to the API produces results that developers can use to execute the next call. This means developers can write a thin layer of API logic to invoke any pebble exposed via the Flow API.

Developers can find API entry point information on the View Pebble panel provided by the Pebble Flow workflow server.

The request and describe API paths are displayed on the developer info panel located on the View Pebble page accessible on the Pebble Flow server.

The 'request' and 'describe' API paths are displayed on the developer info panel on the Pebble Flow server.

The following describes the steps to execute a pebble on the Pebble Flow server.

Health Check

A developer can check for a running server

POST /api/ping

Accept text/plain

curl example

curl --verbose -X POST http://localhost:8080/api/ping

> POST /api/ping HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*

Returns

pong

Retrieve Pebble Metadata

A developer may retrieve extensive metadata that describes the input and output worksheets that make up a pebble. The following example uses curl to retrieve pebble metadata.

GET /api/pebble/[id]/describe

Accept text/csv

Returns data with headers as described in the following table.

headertypedescriptioncomments
nametextName of the worksheetWorksheet names are limited to valid worksheet names in Excel.
classificationtextType of worksheetThe Pebble Stream analyzer will classify worksheets as either input, transient, or output.
headertextName of the column headerThe name of a column header in the worksheet.
typetextType of data in the columnThe types of data in this column. Possible values are :nu (number), :st (text), :dt (Excel date), and :bo (boolean). It is rare, but some columns may have more than one data type defined.

curl example

curl --verbose -X GET http://localhost:8080/api/pebble/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7/describe --header "Accept: text/csv" 

> GET /api/pebble/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7/describe HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: text/csv

Returns 

name,category,header,type
"Capital",inputs,partnership group,:nu
"Capital",inputs,partner id,:nu
"Capital",inputs,amount,:nu
"Taxable Income",inputs,income id,:nu
"Taxable Income",inputs,category,:st
"Taxable Income",inputs,amount,:nu
...
"Allocation Detail",transient,amount,:nu
"Allocation Detail",transient,allocation,:nu
"Partner Allocation",transient,partner id,:nu

The Request Base URI Path

The request base URI path is the common prefix for processing requests to invoke a pebble, check the invocation status, and retrieve its results. The path prefix is viewable in the developer panel of the pebble's detail page and has the form:

/api/request/[request id]

Submit a Pebble Flow Request

The following curl example illustrates how a request can be made to invoke a pebble using the request base URI path. The developer is expected to have examined the pebble's metadata and understood the input necessary to invoke it.

POST /api/request/[request id]

Content-Type multipart/form-data

Accept text/plain

Returns data as described in the following table:

headertypedescriptioncomments
pathstringpath used to determine request statusUse this path to poll the Pebble Flow service for the status of the request specified by id.

curl example

Note the requirement to use multi-part form upload as the conveyor of each worksheet input file. Both "Capital.csv" and "Taxable Income.csv" in the curl request are files containing CSV data.

curl --verbose -F 'Taxable [email protected]' -F '[email protected]' http://localhost:8080/api/request/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7 --header "Accept: text/csv" 
> POST /api/request/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*

Returns:

path
http://localhost:8080/api/request/abe78d0a-4f24-409a-acbd-7d4e83bfcfed/status

Note the returned /api/request/abe78d0a-4f24-409a-acbd-7d4e83bfcfed/status. This URI path is used to determine the request's status via polling.

🚧

Do not assume this path will be the structured the same way for all requests. Do not programmatically construct this path.

Always use the path returned by the request to query for status.

Determine Status

Once a developer submits a request, its request performance status may be periodically checked via polling. The following curl example illustrates how to inspect a request's status.

POST /api/request/[request id]/status

Accept text/csv

Returns data described by the following table:

headertypedescriptioncomments
request-idstringThe request identifierUnique to this request
apistringThe API pathUse this path to retrieve the specified output worksheet
containerstringA location description that is Pebble Flow back end store dependentThis describes the location of where the data is stored. It could be a database table or a cloud store container. It depends on the Pebble Flow deployment. It can be used whenever direct access to Pebble Flow results is required.
pathstringA location description that is specific to the Pebble Flow backend serverThis could be a SQL snippet for database access, a path to a file for a cloud store, or a key to a key-value datastore. It should used whenever direct access to Pebble Flow results is required.

curl example

curl --verbose -X POST http://localhost:8080/api/request/22b3fa80-1786-4381-8daf-d5415d7b2890/status --header "Accept: text/csv"
> POST /api/request/22b3fa80-1786-4381-8daf-d5415d7b2890/status HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: text/csv

Returns

request-id,api,container,path
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/Allocation Detail","requests","192eb621-3505-41af-a330-56e8012762c1/output/Allocation Detail"
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/BOP Ratios","requests","192eb621-3505-41af-a330-56e8012762c1/output/BOP Ratios"
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/Capital Adjustment","requests","192eb621-3505-41af-a330-56e8012762c1/output/Capital Adjustment"
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/Partner Allocation","requests","192eb621-3505-41af-a330-56e8012762c1/output/Partner Allocation"
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/Ratio - Income","requests","192eb621-3505-41af-a330-56e8012762c1/output/Ratio - Income"
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/Validations","requests","192eb621-3505-41af-a330-56e8012762c1/output/Validations"

🚧

If the request expects several output files to be returned, the status call will eagerly return the list of completed outputs.

It is up to the calling party to continually poll to determine the status of the remaining expected outputs.

Receive a Callback

Pebble Flow supports returning status to an endpoint specified during the request invocation. Pebble Flow will provide the same information as polling. The callback endpoint must accept multipart-file upload on an input field named "status." It should also support HTTP Basic Authentication. The HTTP Basic Authentication key should be provided during the request. It is advised that the Basic Authentication key's utility should be finite and short-lived.

During request invocation, provide a fully qualified secure URI to receive status information when your request is completed. The callback URI could be a temporary URI tailored to this request and should require Basic Authentication. The Basic Authentication key to be used should also be provided.

callback parameter example

callback-uri https://example.com/call-us-back

callback-auth-key d8aef8da7f1d461b9bbf3a77bfd0a4cc

curl example

Observe the callback parts and the ability to specify which worksheets should be saved.

curl --verbose -F 'Taxable [email protected]' -F '[email protected]' http://localhost:8080/api/request/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7 -F 'callback-uri=http://localhost:8080/api/test-call-back/123456' -F 'callback-auth=055739d0-20da-4361-93b4-631ffabaae3a' -F "output=Allocation Detail" -F "output=Taxable Income" --header "Accept: text/csv" 
> POST /api/request/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*

Returns:

path
http://localhost:8080/api/request/abe78d0a-4f24-409a-acbd-7d4e83bfcfed/status

Pebble Flow will attempt to execute a call back multiple times if it receives an HTTP status code greater than 399.

All outputs the request was expected to produce are available for download when the callback is executed.

🚧

Pebble Flow will only accept HTTPS callbacks and expects the callback to be made using the POST method.

When calling back, Pebble Flow will set the form parameter statusto contain the status of the request in text/csv format.

Download Results

There are two ways to download results from Pebble Flow. Both mechanisms are explained here. First, developers can programmatically download data using the path URIs returned in the API field of the returned status detail.

download via the Pebble Flow API

GET /api/request/output/[request-id]/

Accept text/csv

Note the format of the file returned is defined in the pebble description. Use the describe API to retrieve that metadata.

curl example

curl --verbose -X GET http://localhost:8080/api/request/22b3fa80-1786-4381-8daf-d5415d7b2890/output/Allocation%20Detail --header "Accept: text/csv" > Allocation-Detail.csv

> GET /api/request/output/22b3fa80-1786-4381-8daf-d5415d7b2890/Allocation%20Detail HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: text/csv
> 

  Returns

partner id,ratio,amount,allocation
4953,0.000679544736138293737476715,-1.0,-0.000679544736138293737476715
4953,0.000679544736138293737476715,1.0,0.000679544736138293737476715
4953,0.000679544736138293737476715,-2.0,-0.00135908947227658747495343
4953,0.000679544736138293737476715,0,0E-27
4953,0.000679544736138293737476715,-2.0,-0.00135908947227658747495343
4953,0.000679544736138293737476715,-0.5,-0.000339772368069146868738358
4953,0.000679544736138293737476715,-1.875,-0.00127414638025930075776884
4953,0.000679544736138293737476715,0,0E-27
4953,0.000679544736138293737476715,2.0,0.00135908947227658747495343

  ... truncated output for example only

direct downloads from the Pebble Flow storage

Developers may retrieve data directly from the Flow internal store to reduce load in the Pebble Flow server or optimize download performance. With this method, the calling service would need access to the Flow datastore. Conveniently, Pebble Flow results are stored as CSV files. The container and path fields in the status detail provide the necessary information to construct the required datastore call. How this is achieved depends on the type of data storage being accessed and is beyond the scope of this documentation.

The container and path information gives the developer enough information to perform direct downloads.

Download Execution Metrics

Developers can access request metrics upon completion of the request. These metrics provide valuable information on request performance.

GET /api/request/[request-id]/metrics

Accept text/csv

Returns data described by the following table:

headertypedescriptioncomments
worksheetstringName of worksheet createdPebble Stream pebbles produce output that maps to the worksheets defined in the compiled spreadsheet.
classificationstringType of worksheet created.Valid values include reference, transient, constant, isolated, and final. Most developers will be concerned with final worksheets as these are output worksheets.
cells computednumberThe number of cells created on completion of this worksheet.To determine the number of rows created, divide this number by the number of columns in the worksheet. This is retrievable using the describe API.
execution timenumberThe time in milliseconds taken to compute this worksheet.This time does not include time to store results.

curl example

The following example uses curl to retrieve request metrics.

curl --verbose -X GET http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/metrics --header "Accept: text/csv"

> GET /api/request/192eb621-3505-41af-a330-56e8012762c1/metrics HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: text/csv
< 
worksheet,classification,cells computed,execution time
"Capital",reference,606,0
"Taxable Income",reference,606,0
"BOP Ratios",transient,606,3
"Ratio - Income",transient,242412,0
"Allocation Detail",transient,161608,421
"Partner Allocation",transient,404,357
"Capital Adjustment",final,808,2
"Validations",final,6,1

Downloading a Pebble for use with the Pebble Stream Runtime

Developers can request the actual pebble file stored by the Pebble Flow server. Using this API, developers may download the Pebble byte file when executing in-memory calls using the embeddable Pebble Stream Runtime.

GET /api/pebble/[uri]/load

Accept application/octet-stream

curl example

curl --verbose -X GET http://localhost:8080/api/pebble/1b27fa15614e6d41d1fb205945cc54a06709e9c79a02165f815f1c95dd01667e/load --header "Accept: application/octets" > pebble-bytes

> GET /api/pebble/1b27fa15614e6d41d1fb205945cc54a06709e9c79a02165f815f1c95dd01667e/load HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: application/octet-stream
> 
< Content-Type: application/octets
< Content-Length: 13954
< Server: Jetty(11.0.18)
< 
{ [13954 bytes data]
100 13954  100 13954    0     0  1288k      0 --:--:-- --:--:-- --:--:-- 1362k
 
 Returned 13954 bytes of application/octets

🚧

Downloading pebbles from Pebble Flow to invoke it via the Pebble Flow Request API is unnecessary.

This feature could be used to dynamically load pebbles from the Pebble Flow server to execute in using the embedded runtime, but is not required to execute pebble on the Pebble Flow server via the API.

General API Notes

Pebble Flow's preferred form of representation is CSV. CSV files offer many advantages over other structured formats like JSON, EDN, XML, and HTML. The CSV file format is relatively lightweight and portable across many language platforms and data stores. It can be streamed and consumed line by line without heavy parsing.

authentication

All API calls are authenticated via HTTP Basic Authentication. The Pebble Flow server API IDs are identified as a configuration parameter.

API error codes

The Pebble Flow API will return the following error codes

400

The request provided to the Flow API was malformed. The reason for this issue will be returned with mime type "text/plain".

401

The request failed authentication

404

The requested resource was not found.

429

The Pebble Flow server is not accepting new requests at the moment. Please wait and retry your request.

500

An unexpected error occurred. Please bring this to your administrator's attention.

numeric formatting

If the precision flag is not set on a process request, Pebble Flow defaults to 24 decimal places of precision.

🚧

Pebble Flow will store some numeric representations in scientific notation in order to optimize performance.

Make sure to use a CSV reader that understands how to deal with scientific notation.

date representation

Pebble Stream processes dates using Excel's numeric representation. For columns with type :dt (date), convert UNIX timestamps or date strings to Excel's numeric date representation.

input worksheet column ordering

For Input worksheets, Pebble Flow will require all of the headers defined in the worksheet to be present in order to begin execution. However, if the headers are present in an order different from the worksheet in the pebble definition, Pebble Flow will automatically place the header columns in the expected order.

Developers should regard the precision requested of Pebble Flow for a given request and guard against loss of precision whenever possible.

file compression during transport.

Pebble Flow will apply HTTP gzip compression whenever possible.