TITLE: Flow diagram for data cleaning process
DATE: 2020-06-10
AUTHOR: John L. Godlee
====================================================================
I've been involved in improving the data cleaning process for the
SEOSAW database. I created an R package with a load of data
cleaning functions and designed a workflow for ingesting new data
into the database. Today I faffed around for a while making a flow
diagram to keep track of all the functions.
[SEOSAW]:
https://seosaw.github.io/
I opted for a nested function design. At the bottom end there are
many functions which do very simple things like check whether a
particular column contains the correct factor levels (e.g. diam()).
These functions are the nested within a function which checks all
of the column contents in a dataset which is to be ingested into
the database (e.g. colValCheck()). At the top there are a small
number of high level functions which perform the checking
formatting and adding new columns all in one (e.g. stemTableGen()).
I'm not sure whether this is overly complicated or not, but to me
it seems reasonably intuitive, made easier with some good
documentation. It also affords the user a lot of flexibility in how
they construct their workflow.
The idea is the diagram will be hosted on the SEOSAW website along
with a vignette and the package manual, as well as on the Bitbucket
repository for the SEOSAW dataset, so that users can use it as a
quick reference when cleaning their own data, either for inclusion
into the SEOSAW database, or for comparing their own data with the
SEOSAW repository.
