Data Flow in Lab and Field

Our Sci
1

Hi all

@DanT and I had a very good session today where we had some hands on experience with dashboards.

While talking about how samples will be processed in the lab a couple of topics were arising.

Calibration

Before a batch of samples is processed, calibration is necessary where linear regression is used to create a fit. One option would be to hardcode the calibration then inside the measurement script. The more ideal option would be to do a survey for the calibration, pair it with an ID and then fetch the calibration during the measurement from the web application (via JSON or CSV). The ID of the calibration is entered as text question inside the measurement survey.

Micro Surveys vs Large Surveys

In General most surveys will consist of an ID which identifies the sample or the sample location and a series of measurements.

In the lab, the flow is to run a certain measurement for every single sample. This has the effect to leave many surveys open. Finding the proper survey would require a search function where partially answered surveys are queried for a specific sample ID.

Another option would be to create micro surveys, where the survey would only consist of a single measurement and ID. Then in the dashboard, all data would be pulled together and joined on the sample ID.

Benefits of micro surveys

  • survey is done after single measurement
  • integrity of data could be better (if something goes wrong, with the entire data set, only single measurement is affected)
  • possibly assign multiple measurements for single ID may be helpful
  • allows service technician to complete simple survey without lab needing to continue started survey
  • could benefit comparability of data (?) or the opposite (?)

Benefits of long surveys

  • clearly shows user the progress within a survey
  • strongly coherent data sets
  • clear indication when survey is completed
  • fewer surveys

Both options have their benefits. For downloading a single CSV currently the long surveys are the only option, yet building a dashboard that merges multiple surveys into a single CSV would not be too difficult to implement.

A mix of both options would allow service technicians in the field to have a longer survey with the benefit of fewer surveys and clear indication of progress, and the lab to split the rest into small surveys for maximum efficiency.

Conclusion

This is a topic which we need to discuss further to come to a clear conclusion, the good news is that both methods are possible to do in practice with the current setup. For efficiency the only feature currently missing in the Android application is the option to search survey for an ID oder generic content.

2

@mdc Dan and I discussed this some last week also…Now that we have some experience in the lab, makes most sense is to create 3 surveys:

  1. Metadata - samples comes in, you scan barcode and enter metadata (if submitted in paper form). This is the ‘intake’ step to ensure nothing that comes in gets missed. Data Partners can push this survey directly from the field.
  2. Food - all food data
  3. Soil - all soil data

In actual use cases, a phone may have 100 or more partially completed surveys and 100s of completed surveys in it at any given time. This is important to know in terms of performance and stability.

The key to making this actually workable in the lab is:

  1. Barcode search - I think we’ve talked this to death, but here it is again :slight_smile: You can take an image of a barcode, and find all saved surveys (partial surveys included) which contain that barcode.
  2. Stored surveys are sorted in the display by 1) partial surveys on top and 2) by the time they were entered.
  3. When you click on a partial survey from search, you immediately go to next question in the survey which is unaswered. If the survey is completed, you go to the overview screen for that survey (which shows all question with the checkboxes on the right hand side).
  4. Lab SOP for parts of the process which involve lots manual data input (weights, or things like that), we will have a manifest with spaces to put write in each weight next to the barcode. The manifest will travel with the sample, get filled out by pen along the way, and at the end of the process will be entered in all at once into the survey.
  5. Barcodes should be relatively short (less than 5 numbers/letters) but be able to represent a large number of unique samples. Using 0-9 and A - Z will allow a very large set of unique combinations with only 5 number/letters (45M combinations should be good).