Pebble Flow Workflow Server
The Pebble Flow Server is a private cloud deployable server that can expose pebbles as services via RESTful APIs.
Deploying Pebble Flow on Microsoft's Azure Cloud
Create the resource
The Pebble Flow Workflow Server is deployable as an Azure resource. It is available on the Azure Marketplace. Click on "Create a resource"
Azure services's "Create a Resource" button
Search for the Pebble Flow Workflow Server
Click on the Pebble Flow Workflow Server card.
Search for "Pebble Stream" to find the Pebble Flow server on Azure Marketplace.
Begin Pebble Flow deployment
Point-and-click deployment via the Azure Marketplace UI is straightforward and completed in three steps. Choose your appropriate plan, then click the blue "Create" button to start configuring the Pebble Flow deployment.
Click on the 'Create' button to begin Azure deployment.
Choose the plan that best fits your organization's needs.
Step 1 - select your deployment subscription and choose resource names
In the first step, select your subscription and a unique resource group for deployment. You should isolate Pebble Flow's deployment from your other Azure resource groups. Choose your App Service plan and select a storage account type that meets your business continuity need.
$186.15/
Choose your subscription, then select names for the deployable resource group.
Be sure to choose plans that support autoscaling if you want to use that feature.
Step 2 - Enter the API key and specify the admin user
Enter your API key (contact us to get this), along with the email address and name of the Pebble Flow administrator. The deployment will automatically create an account for this user. Make sure to enter the correct email address at this step.
Please enter the API key, a valid admin user email and name, and the organization's name.
Step 3 - optionally copy metadata from a previous installation.
If there was a previous Pebble Flow installation, you may copy its pebble resources to this new installation. If you don't want to do this, uncheck "Copy data from existing storage account" and proceed to the next step.
Copy users, compiled pebbles, spreadsheets, workflows, authorizations, and API endpoints to the new installation.
Request, job, and scenario results from the previous install are not copied.
Review configuration, accept terms, then deploy
On the final screen, you must agree to the terms and conditions for using Pebble Flow on the Azure Marketplace.
Agree to the terms and conditions for running resources procured from Azure's Marketplace.
$186.15/The Pebble Flow Workflow Server deployment should be completed in 30 minutes or less.
Autoscaling Pebble Flow on Microsoft's Azure Cloud
Pebble Flow is designed to run as a dynamically distributed service on Azure. Azure App Service's built-in autoscaling can be used to manage costs. Pebble Flow's API uses Azure Blobstore's queues to manage request traffic, and Azure App Service is designed to monitor queue traffic for scaling cues.
Configuring Azure App Service autoscaling via the portal is simple.
You can use Azure's portal to configure scaling rules based on pending Pebble Flow request message counts.
One way to configure your scaling capability using Azure's portal is to scale up the number of Pebble Flow servers as long as the average number of pending requests is above a certain threshold. Scaling down could be triggered by completely draining the request queue. Of course, the maximum number of servers can be capped, regardless of the number of pending requests.
Make sure to choose an App Service sku that supports autoscaling. If you don't, you can always change your plan.
The Pebble Flow API
Invoking a Pebble via the Pebble Flow Service
Whenever a pebble is compiled, the Pebble Flow server exposes the pebble to the enterprise via a regular HTTPS interface. The Pebble Flow API is hypermedia-driven; each call to the API produces results that developers can use to execute the next call. This means developers can write a thin layer of API logic to invoke any pebble exposed via the Flow API.
Developers can find API entry point information on the View Pebble panel provided by the Pebble Flow workflow server.
The 'request' and 'describe' API paths are displayed on the developer info panel on the Pebble Flow server.
The following describes the steps to execute a pebble on the Pebble Flow server.
Health Check
A developer can check for a running server
POST /api/ping
Accept text/plain
curl example
curl --verbose -X POST http://localhost:8080/api/ping
> POST /api/ping HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*
Returns
pong
Retrieve Pebble Metadata
A developer may retrieve extensive metadata that describes the input and output worksheets that make up a pebble. The following example uses curl to retrieve pebble metadata.
GET /api/pebble/[id]/describe
Accept text/csv
Returns data with headers as described in the following table.
| header | type | description | comments |
|---|---|---|---|
| name | text | Name of the worksheet | Worksheet names are limited to valid worksheet names in Excel. |
| classification | text | Type of worksheet | The Pebble Stream analyzer will classify worksheets as either input, transient, or output. |
| header | text | Name of the column header | The name of a column header in the worksheet. |
| type | text | Type of data in the column | The types of data in this column. Possible values are :nu (number), :st (text), :dt (Excel date), and :bo (boolean). It is rare, but some columns may have more than one data type defined. |
curl example
curl --verbose -X GET http://localhost:8080/api/pebble/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7/describe --header "Accept: text/csv"
> GET /api/pebble/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7/describe HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: text/csv
Returns
name,category,header,type
"Capital",inputs,partnership group,:nu
"Capital",inputs,partner id,:nu
"Capital",inputs,amount,:nu
"Taxable Income",inputs,income id,:nu
"Taxable Income",inputs,category,:st
"Taxable Income",inputs,amount,:nu
...
"Allocation Detail",transient,amount,:nu
"Allocation Detail",transient,allocation,:nu
"Partner Allocation",transient,partner id,:nu
The Request Base URI Path
The request base URI path is the common prefix for processing requests to invoke a pebble, check the invocation status, and retrieve its results. The path prefix is viewable in the developer panel of the pebble's detail page and has the form:
/api/request/[request id]
Submit a Pebble Flow Request
The following curl example illustrates how a request can be made to invoke a pebble using the request base URI path. The developer is expected to have examined the pebble's metadata and understood the input necessary to invoke it.
POST /api/request/[request id]
Content-Type multipart/form-data
Accept text/plain
Returns data as described in the following table:
| header | type | description | comments |
|---|---|---|---|
| path | string | path used to determine request status | Use this path to poll the Pebble Flow service for the status of the request specified by id. |
curl example
Note the requirement to use multi-part form upload as the conveyor of each worksheet input file. Both "Capital.csv" and "Taxable Income.csv" in the curl request are files containing CSV data.
curl --verbose -F 'Taxable [email protected]' -F '[email protected]' http://localhost:8080/api/request/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7 --header "Accept: text/csv"
> POST /api/request/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*
Returns:
path
http://localhost:8080/api/request/abe78d0a-4f24-409a-acbd-7d4e83bfcfed/status
Note the returned /api/request/abe78d0a-4f24-409a-acbd-7d4e83bfcfed/status. This URI path is used to determine the request's status via polling.
Do not assume this path will be the structured the same way for all requests. Do not programmatically construct this path.
Always use the path returned by the request to query for status.
Determine Status
Once a developer submits a request, its request performance status may be periodically checked via polling. The following curl example illustrates how to inspect a request's status.
POST /api/request/[request id]/status
Accept text/csv
Returns data described by the following table:
| header | type | description | comments |
|---|---|---|---|
| request-id | string | The request identifier | Unique to this request |
| api | string | The API path | Use this path to retrieve the specified output worksheet |
| container | string | A location description that is Pebble Flow back end store dependent | This describes the location of where the data is stored. It could be a database table or a cloud store container. It depends on the Pebble Flow deployment. It can be used whenever direct access to Pebble Flow results is required. |
| path | string | A location description that is specific to the Pebble Flow backend server | This could be a SQL snippet for database access, a path to a file for a cloud store, or a key to a key-value datastore. It should used whenever direct access to Pebble Flow results is required. |
curl example
curl --verbose -X POST http://localhost:8080/api/request/22b3fa80-1786-4381-8daf-d5415d7b2890/status --header "Accept: text/csv"
> POST /api/request/22b3fa80-1786-4381-8daf-d5415d7b2890/status HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: text/csv
Returns
request-id,api,container,path
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/Allocation Detail","requests","192eb621-3505-41af-a330-56e8012762c1/output/Allocation Detail"
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/BOP Ratios","requests","192eb621-3505-41af-a330-56e8012762c1/output/BOP Ratios"
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/Capital Adjustment","requests","192eb621-3505-41af-a330-56e8012762c1/output/Capital Adjustment"
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/Partner Allocation","requests","192eb621-3505-41af-a330-56e8012762c1/output/Partner Allocation"
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/Ratio - Income","requests","192eb621-3505-41af-a330-56e8012762c1/output/Ratio - Income"
192eb621-3505-41af-a330-56e8012762c1,"http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/output/Validations","requests","192eb621-3505-41af-a330-56e8012762c1/output/Validations"
If the request expects several output files to be returned, the status call will eagerly return the list of completed outputs.
It is up to the calling party to continually poll to determine the status of the remaining expected outputs.
Receive a Callback
Pebble Flow supports returning status to an endpoint specified during the request invocation. Pebble Flow will provide the same information as polling. The callback endpoint must accept multipart-file upload on an input field named "status." It should also support HTTP Basic Authentication. The HTTP Basic Authentication key should be provided during the request. It is advised that the Basic Authentication key's utility should be finite and short-lived.
During request invocation, provide a fully qualified secure URI to receive status information when your request is completed. The callback URI could be a temporary URI tailored to this request and should require Basic Authentication. The Basic Authentication key to be used should also be provided.
callback parameter example
callback-uri https://example.com/call-us-back
callback-auth-key d8aef8da7f1d461b9bbf3a77bfd0a4cc
curl example
Observe the callback parts and the ability to specify which worksheets should be saved.
curl --verbose -F 'Taxable [email protected]' -F '[email protected]' http://localhost:8080/api/request/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7 -F 'callback-uri=http://localhost:8080/api/test-call-back/123456' -F 'callback-auth=055739d0-20da-4361-93b4-631ffabaae3a' -F "output=Allocation Detail" -F "output=Taxable Income" --header "Accept: text/csv"
> POST /api/request/68c2f761b64f0a62199787c76a6befa09fd8ce3961ee65aa85b017e593cc82f7 HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: */*
Returns:
path
http://localhost:8080/api/request/abe78d0a-4f24-409a-acbd-7d4e83bfcfed/status
Pebble Flow will attempt to execute a call back multiple times if it receives an HTTP status code greater than 399.
All outputs the request was expected to produce are available for download when the callback is executed.
Pebble Flow will only accept HTTPS callbacks and expects the callback to be made using the POST method.
When calling back, Pebble Flow will set the form parameter
statusto contain the status of the request in text/csv format.
Download Results
There are two ways to download results from Pebble Flow. Both mechanisms are explained here. First, developers can programmatically download data using the path URIs returned in the API field of the returned status detail.
download via the Pebble Flow API
GET /api/request/output/[request-id]/
Accept text/csv
Note the format of the file returned is defined in the pebble description. Use the describe API to retrieve that metadata.
curl example
curl --verbose -X GET http://localhost:8080/api/request/22b3fa80-1786-4381-8daf-d5415d7b2890/output/Allocation%20Detail --header "Accept: text/csv" > Allocation-Detail.csv
> GET /api/request/output/22b3fa80-1786-4381-8daf-d5415d7b2890/Allocation%20Detail HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: text/csv
>
Returns
partner id,ratio,amount,allocation
4953,0.000679544736138293737476715,-1.0,-0.000679544736138293737476715
4953,0.000679544736138293737476715,1.0,0.000679544736138293737476715
4953,0.000679544736138293737476715,-2.0,-0.00135908947227658747495343
4953,0.000679544736138293737476715,0,0E-27
4953,0.000679544736138293737476715,-2.0,-0.00135908947227658747495343
4953,0.000679544736138293737476715,-0.5,-0.000339772368069146868738358
4953,0.000679544736138293737476715,-1.875,-0.00127414638025930075776884
4953,0.000679544736138293737476715,0,0E-27
4953,0.000679544736138293737476715,2.0,0.00135908947227658747495343
... truncated output for example only
direct downloads from the Pebble Flow storage
Developers may retrieve data directly from the Flow internal store to reduce load in the Pebble Flow server or optimize download performance. With this method, the calling service would need access to the Flow datastore. Conveniently, Pebble Flow results are stored as CSV files. The container and path fields in the status detail provide the necessary information to construct the required datastore call. How this is achieved depends on the type of data storage being accessed and is beyond the scope of this documentation.
The container and path information gives the developer enough information to perform direct downloads.
Download Execution Metrics
Developers can access request metrics upon completion of the request. These metrics provide valuable information on request performance.
GET /api/request/[request-id]/metrics
Accept text/csv
Returns data described by the following table:
| header | type | description | comments |
|---|---|---|---|
| worksheet | string | Name of worksheet created | Pebble Stream pebbles produce output that maps to the worksheets defined in the compiled spreadsheet. |
| classification | string | Type of worksheet created. | Valid values include reference, transient, constant, isolated, and final. Most developers will be concerned with final worksheets as these are output worksheets. |
| cells computed | number | The number of cells created on completion of this worksheet. | To determine the number of rows created, divide this number by the number of columns in the worksheet. This is retrievable using the describe API. |
| execution time | number | The time in milliseconds taken to compute this worksheet. | This time does not include time to store results. |
curl example
The following example uses curl to retrieve request metrics.
curl --verbose -X GET http://localhost:8080/api/request/192eb621-3505-41af-a330-56e8012762c1/metrics --header "Accept: text/csv"
> GET /api/request/192eb621-3505-41af-a330-56e8012762c1/metrics HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: text/csv
<
worksheet,classification,cells computed,execution time
"Capital",reference,606,0
"Taxable Income",reference,606,0
"BOP Ratios",transient,606,3
"Ratio - Income",transient,242412,0
"Allocation Detail",transient,161608,421
"Partner Allocation",transient,404,357
"Capital Adjustment",final,808,2
"Validations",final,6,1
Downloading a Pebble for use with the Pebble Stream Runtime
Developers can request the actual pebble file stored by the Pebble Flow server. Using this API, developers may download the Pebble byte file when executing in-memory calls using the embeddable Pebble Stream Runtime.
GET /api/pebble/[uri]/load
Accept application/octet-stream
curl example
curl --verbose -X GET http://localhost:8080/api/pebble/1b27fa15614e6d41d1fb205945cc54a06709e9c79a02165f815f1c95dd01667e/load --header "Accept: application/octets" > pebble-bytes
> GET /api/pebble/1b27fa15614e6d41d1fb205945cc54a06709e9c79a02165f815f1c95dd01667e/load HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.81.0
> Accept: application/octet-stream
>
< Content-Type: application/octets
< Content-Length: 13954
< Server: Jetty(11.0.18)
<
{ [13954 bytes data]
100 13954 100 13954 0 0 1288k 0 --:--:-- --:--:-- --:--:-- 1362k
Returned 13954 bytes of application/octets
Downloading pebbles from Pebble Flow to invoke it via the Pebble Flow Request API is unnecessary.
This feature could be used to dynamically load pebbles from the Pebble Flow server to execute in using the embedded runtime, but is not required to execute pebble on the Pebble Flow server via the API.
General API Notes
Pebble Flow's preferred form of representation is CSV. CSV files offer many advantages over other structured formats like JSON, EDN, XML, and HTML. The CSV file format is relatively lightweight and portable across many language platforms and data stores. It can be streamed and consumed line by line without heavy parsing.
authentication
All API calls are authenticated via HTTP Basic Authentication. The Pebble Flow server API IDs are identified as a configuration parameter.
API error codes
The Pebble Flow API will return the following error codes
400
The request provided to the Flow API was malformed. The reason for this issue will be returned with mime type "text/plain".
401
The request failed authentication
404
The requested resource was not found.
429
The Pebble Flow server is not accepting new requests at the moment. Please wait and retry your request.
500
An unexpected error occurred. Please bring this to your administrator's attention.
numeric formatting
If the precision flag is not set on a process request, Pebble Flow defaults to 24 decimal places of precision.
Pebble Flow will store some numeric representations in scientific notation in order to optimize performance.
Make sure to use a CSV reader that understands how to deal with scientific notation.
date representation
Pebble Stream processes dates using Excel's numeric representation. For columns with type :dt (date), convert UNIX timestamps or date strings to Excel's numeric date representation.
input worksheet column ordering
For Input worksheets, Pebble Flow will require all of the headers defined in the worksheet to be present in order to begin execution. However, if the headers are present in an order different from the worksheet in the pebble definition, Pebble Flow will automatically place the header columns in the expected order.
Developers should regard the precision requested of Pebble Flow for a given request and guard against loss of precision whenever possible.
file compression during transport.
Pebble Flow will apply HTTP gzip compression whenever possible.
Updated about 1 month ago