Adoption and Deployment of Flow Interoperability Standards - Request for Applications

Adoption and Deployment of Flow Interoperability Standards

Flow Results support for Turn.io

Submitted by Lieze Langford (Turn.io) on Fri, 01/15/2021 - 05:49
Application Status: 
Concept Note co-creation
Executive Summary: 

This investment will go towards the development of the necessary API functionality to allow for data captured to be shared off platform.

The larger goal is to facilitate data sharing between Turn.io and the larger technology ecosystem.

Turn.io implements and exposes the WhatsApp Business API, however, the WhatsApp Business API does not expose any functionality for exporting of captured data during a conversation, this needs to be implemented by Turn.io.

Rather than inventing our own proprietary format to address this problem, we want to align with the larger international development community to facilitate interoperability.

The concept of a flow is already implemented as a feature called Threads in the Turn.io application.

Thus the Flow Results support will export the data captured in Turn.io Threads as a Flow Results data package.

Two-sentence Overview: 

Turn.io allows for social impact teams to manage conversations and capture data using the WhatsApp Business API at scale.

Data is already being captured and stored on the Turn.io platform, making the data available as a Flow Results data package allows for interoperability with the larger international development and tech ecosystem. 

Consortium team: 

The work will be added to the Turn.io roadmap and implemented by the Turn.io product team. The team works with weekly iterations and larger feature work is allocated per quarters.

If the grant is approved, the work will be scheduled in the current or next up quarter.

The work will be led by Turn.io’s product manager.

Supporting organisations are some of Turn.io’s key clients, including the WHO, The South African Department of Health, and Praekelt Foundation.

The WhatsApp covid-19 services offered by the WHO and The South African Department of Health are among the largest WhatsApp Business API installations in the world.

Project Description: 

Problem statement:

Many social impact organisations are using Turn.io to capture data using Threads, Turn.io’s implementation of the Flow concept.

Currently, Turn.io allows for data to be exported off platform in real-time using webhooks or via a data lake integration to BigQuery.

Both mechanisms are powerful but require a relatively high level of technical expertise to use effectively and there is no easy way to export data captured using Turn.io threads to allow interoperability with the larger international development technology sector.

Objectives:

Allow for data captured with Turn.io threads to be exported as a Flow Results data package by:

  1. Exposing an HTTP API endpoint per Turn.io Thread which returns the data captured in the given Thread as a paginated Flow Results data package.
  2. Allowing the UI to download a Flow Results data package as a zip file combining the resource description and the resource data. This will build on the API but not require the Turn.io user to worry about pagination.

Deliverables:

  1. An open-source Flow Results Data Package library for the Elixir programming language hosted on hex.pm and https://github.com/turnhub licensed under an OSI recognised license.
  2. Published documentation on Turn.io’s support for Flow Results on https://learn.turn.io
  3. Published documentation on Turn.io’s API support for Flow Results on https://whatsapp.turn.io/docs 
  4. Support in Turn.io’s Thread builder UI for downloading the captured data as a Flow Results Data Package.

Schedules:

  • We expect this work to take 2 to 3 weeks from start to a first beta release exposed as a feature flag in the Turn.io UI.
  • Work will start depending on the time of approval and work allocation at the time, and expect it to be scheduled within a calendar quarter.

Risks & Mitigations:

Many of the clients using Turn.io run at significant scale in terms of uniques. One risk is that the approach of generating a ZIP file does not scale for very large initiatives.

If that is the case then the API is a possible fallback and a mitigation is to then restrict the ZIP file based approach to a to be determined maximum number of records.

 

 

Comments

Mark Boots - Mon, 01/25/2021 - 20:45

This concept note demonstrates strong understanding and alignment with the Flow Results goals. As outcomes, it would enable Turn.io threads to sit alongside other FR-compliant channels at the "messaging edge" of core data systems. It would also allow Turn.io thread interactions to be visualized or archived by FR standards-based consumers, adding value to the overall FR ecosystem. 

Two questions on technical implementation/feasibility:

  1. How do Turn.io Threads handle changes to the thread content, once some initial responses have been collected? If a Thread can be modified after collecting data, would there be a feasibl way to provide the "Option 2" behaviour described in https://floip.gitbook.io/flow-results-specification/specification#result... ? (This often depends on being able to associate responses internally with the "iteration" of the thread that collected them.) Or would it be necessary to treat each "iteration" of a Thread as a separate Flow Results data package?
  2. Is there a feasible mapping of Thread interaction types to the Flow Results 'question' types? Are there any new 'question' types that should be suggested for the Flow Results spec in order to make this possible?

Thanks!

Lieze Langford - Thu, 01/28/2021 - 11:00

 1)  Responses of users that started thread version A, but are still busy completing the thread even though the updated version B might have been created in the interim, are still recorded for the thread definition of version A.

So yes it is versioned, and responses are recorded against a specific version, and thus we have enough information to be able to make versioned exports.

Turn.io may be able to support this but not in the initial version, keeping in mind that thread data export has not yet been implemented.


2) Threads currently support text questions (which accept any response) or multiple-choice questions (which accept one of a list), these are currently adequately covered by the flow results spec.