September 27, 2016 at 11:41 am #10510SteveParticipant
I think soon machine learning/AI will produce technology that we don’t understand, it will work and work better than what humans can do, but we won’t understand how it works.
This could relate to new chip designs (which are mainly rules based), material science, etc.September 27, 2016 at 9:32 pm #10576JiefeiParticipant
We (humans) don’t fully understand how deep learning (e.g., CNN) works. but thanks to it machine can now do better than humans in image recognition.
Not sure when we will fully understand deep neural nets.October 6, 2016 at 2:10 pm #10869PaulKeymasterOctober 9, 2016 at 3:03 am #10933JiefeiParticipant
@paulharrison, no, we are still far from fully understand how deep neural nets work (or should work). Take CNN as an example, we may know how convolution works, we may know how pooling works, we may also know how gradient descent works. But no one can answer, given a data set, how many kernels we need, what should be the right kernel sizes, how many layers we should build, and etc. Everything is trial-and-error (although past experience may help). There is no mathematical theory backing it. In contrast, think about building a space rocket, each part of the design can be and must be precisely and mathematically calculated.
In fact, this is a well-known criticism of deep learning in the academic community.October 10, 2016 at 1:12 pm #10947PaulKeymaster
@jiefeima I see what you mean, although that argument is equally valid for all supervised machine learning methods, not just deep learning; one always takes the same trial and error approach. Are there closed form solutions for any hyperparameter optimisation problems?January 7, 2017 at 2:39 pm #12541Istvan MakareszParticipant
Oh, it seems to be the right place for asking a few questions. (you probably already have provided the answers). I am very interested in this network complexity topic from a biological perspective. In some early work on CNN (Churchland) they found what I guess is the essence of what you debated above:namely given a task (it was differentiating sonic echoes of underwater mines from those of rocks) too simple systems couldn’t even get good results on training sets (Churchland called them ‘dumb’), while overly complicated systems developed dedicated paths on the training sets and then performed poorly on’ natural’ samples (he called these ‘lazy’). So is it still a prevalent problem in CNN design nowadays and there is no way to predict necessary complexity?
You must be logged in to reply to this topic.