What is this about?

Do you remember that garbage in, garbage out?

This is the most important meaning of data cleaning process, which stands for identifying errors in data and repairing them. Here are some tips for cleaning data in the most effective way from my workday routine.

*My specialty is Computer Vision data management, so my examples mostly come from that area

Backup before starting

Make a backup copy of data to be cleaned. Remember Murphy's Law and be a little paranoid to prevent very painful data losses.

From my experience:

• not closing properly JSON file might caused to its erasing;

• file renaming did not work as expected

Use relevant metadata

Very important to understand mission of the data to be preparing and depends on it, select only relevant data fields.

It often happens that we collected more metadata than required. On the one hand, more metadata brings more openness. On the other hand, we are pulling metadata during whole pipe, that goes unused (but still need to be stored, validated etc)

From my experience:

• many times I am creating “shrinker” metadata JSONs from original;

• for efficient images processing especially crucial to consider both the size of the images and the size of the container (folder, Docker, etc.) where the images are stored. For instance, in a project where both RGB and depth images were collected, if the existing pipeline was unable to process one type of image, I prefer to proceed with type required at that moment and optimize processing by half.

Explore the data

It’s not a new idea that the cost of fixing a bug increases exponentially the later in development that bug is discovered.

Same with data cleansing! The issues detected in raw records or the errors caught on early stages of data preparation might prevent a double work and ensure high quality.

On those stages validation should be simple, but open-minded (and not always automated). It can be enough to open a few random metadata files, pay attention to file sizes, and review metadata.

If you have requirements or know which metadata fields are most required/needed, you can ensure that the data contains those fields in the required format.

I am usually working with images/videos, so my checklist:

• [ ] Images are not corrupted, video is playable

• [ ] Configuration and metadata files existed

• [ ] Configuration and metadata files are in correct format and data types

• [ ] Resolution matches the images/videos real resolution

• [ ] Size of images/videos matches their resolution

This early exploration of data does not replace the validation pipeline but provides assurance that crucial fields will not become problematic in later stages.

No place for empties

Empties - there are metadata fields which contained values such as: None, N\A, NaN.

There are two possible reasons for empties:

• noisy bug that erased actual field values;

• metadata fields that are rarely used or not used at all (fields that are no longer in use, designed for future use and required for pipeline support).

Empties of the second type required memory, validation coverage, but not going to be used. They are redundancy and might be excluded.

From my experience:

• empties could be fieIds’ which will be re-annotated in later stages of data processing

• no new empties created after data processing (copying, generating etc)

Do not clean the corners

During data cleaning, we often encounter issues, errors, or even bugs. Those records can cause significant delays in further data processing.

To avoid this, perform quick analysis of the amount those record and errors nature. Depending on the number, continue investigating of the error source or set aside the corrupted records temporarily and proceed with the clean data.

For instance, consider a batch containing 100 records. During the initial validation, 2 records have missing fields. In a later validation, 5 more records encounter errors. The remaining 93 records successfully pass all validations. Once 93% of data is processed, can concentrated with those 7%. In the worst-case scenario, if these records cannot be resolved, they can be excluded from the batch, the main batch is ready by the required deadlines.

Of course, behavior should be different if 50 or 70 % of records are corrupted.

From my experience:

Once I discover that some records are corrupted, I notify the relevant colleagues about the possibility of splitting the batch into two. This allows them to be prepared and plan accordingly (e.g., splitting for validation/training sets).

Take a bite

Before starting large batch processing, run it on a few records to ensure there are no unexpected issues. If any issues are discovered, plan your time and deadlines accordingly.

Visualize

While working with large datasets, such as images or videos, it can be difficult to summarize all parts, especially in JSON or CSV rows. Visualizing metadata can help in understanding and covering the data more comprehensively.

Clean data is a sign of a quality product!