AI data collection methods & techniques: Four approaches for effective algorithm training

Use open source datasets

There are several sources of open source datasets that can be used to train machine learning algorithms, including Kaggle, Data.Gov and others. These datasets give you quick access to large volumes of data that can help to get your AI projects off the ground. But while these datasets can save time and reduce the cost involved with custom data collections, there are other factors to consider. First is relevancy; you need to make sure that the dataset has enough examples of data that is relevant to your specific use case. Second is reliability; understanding how the data was collected and any bias it might contain is very important when determining whether it should be used for your AI project. Finally, the security and privacy of the dataset must also be evaluated; be sure to perform due diligence in sourcing datasets from a third-party vendor that uses strong security measures and shows compliance with data privacy guidelines such as GDPR and the California Consumer Privacy Act.

Generate synthetic data

Rather than collecting real-world data, companies can use a synthetic dataset, which is based upon an original dataset, but then elaborated upon. Synthetic datasets are designed to have the same characteristics as the original, without the inconsistencies (though the potential lack of probabilistic outliers may lead to datasets that don’t capture the full nature of the problem you’re trying to solve). For companies facing strict security, privacy and retention guidelines, such as healthcare/pharma, telco and financial services, synthetic datasets may be a great path toward developing your AI experience.

Export data from one algorithm to another

Otherwise known as transfer learning, this method of collecting data involves using a pre-existing algorithm as the foundation for training a new algorithm. There are clear benefits to this approach in that it can save time and money, but it will only work when advancing from a general algorithm or operational context, to one that’s more specific in nature. Common scenarios in which transfer learning is used include: natural language processing that uses written text, and predictive modeling that uses either video or still photography. Many photo management apps, for example, use transfer learning as a way of creating filters for friends and family members, so you can quickly locate all of the pictures someone appears in.

Collect primary/custom data

Sometimes the best foundation for training a ML algorithm involves collecting raw data from the field that meets your particular requirements. Loosely defined, this could include scraping data from the web, but it could go so far as developing a bespoke program for capturing images or other data in the field. And depending on the type of data needed, you could either crowdsource the collection process, or work with a qualified engineer who knows the ins and outs of collecting clean data (thus minimizing the amount of post-collection processing). Types of data that are collected can range from video and still imagery to audio, human gestures, handwriting, speech, or text utterances. Investing in a custom data collection to generate data that best fits your use case can take more time than using an open source dataset, but the benefits in terms of accuracy, reliability, privacy and bias reduction make this a worthwhile investment.