When dealing with high dimensional data computers have already beaten humans in detecting of complex patterns in some problems. In some other aspects such as being able to ‘read’ human behaviour while crossing road (especially in self driving cars), we need to make cars even smarter (but that’s because we humans cam be quite random at times). 🙂
This is a great question. Worlds open when thinking about it. In a big dataset there are endless patterns that can be found, but most of them are meaningless. What distinguishes the worthwhile ones from the useless. One idea is ‘compressability’. Patterns that allow me to express the data in shorter form are worthwhile; but it gets really tricky to define compressability in a way that can actually be used. Another idea is that useful patterns allow prediction, so if I spot some aspect of the dataset that allows me to predict some other aspect of the dataset that’s a good thing. This is a really important part of AI research right now, called self-supervised learning.
Comments
Lewis commented on :
This is a great question. Worlds open when thinking about it. In a big dataset there are endless patterns that can be found, but most of them are meaningless. What distinguishes the worthwhile ones from the useless. One idea is ‘compressability’. Patterns that allow me to express the data in shorter form are worthwhile; but it gets really tricky to define compressability in a way that can actually be used. Another idea is that useful patterns allow prediction, so if I spot some aspect of the dataset that allows me to predict some other aspect of the dataset that’s a good thing. This is a really important part of AI research right now, called self-supervised learning.