 |
Post ASnnyhPNizoWo1xEqu by [email protected] |
 |
More posts by [email protected] |
|
Post #ASnmqxlWvQHlGZ3s3M by [email protected] |
|
0 likes, 0 repeats |
|
|
|
having some success with a soft ReLU function that can be morphed to sharp ReLU… |
|
|
|
Post #ASnnSj8VlUeKtkomO0 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@lritter Is that common? Would it not undermine or change the stability of how … |
|
|
|
Post #ASnnSjD7UMKt82ySZM by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@nick this is original research afaik. training of my small network is able to … |
|
|
|
Post #ASnnyhPNizoWo1xEqu by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@lritter got it, that’s fascinating |
|
|
|
Post #ASnnyhULQXmf3QHCaW by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@nick but even the soft ReLU function has trouble approximating some curves - f… |
|
|
|
Post #ASnoBahlxWHDmRhhwm by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@lritter nice idea since it can be replaced with (much cheaper) ReLU at inferen… |
|
|
|
Post #ASnoBan5dkWw2wBxEe by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@baldand morphing/switching between similar LU functions could help with a lot … |
|
|
|
Post #ASnpLNDB59VgmUcciG by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@lritter have you tried varying learning rate during training? That is quite co… |
|
|
|
Post #ASnpLNHQpKuezgc1LM by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@baldand even with a low training rate, my small network failed to converge on… |
|
|
|
Post #ASnyS8oltXsjkqVZ6u by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@rich i'm doing all this in a handwritten shader on 1D examples; if we want… |
|
|
|
Post #ASnz3zoZohGwptT39U by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@rich so far it's evident that the choice of activation function directly i… |
|
|
|
Post #ASoTdoCPTgY1rQblwW by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@lritter @rich What if you have it learn the sharpness too? I remember some pa… |
|
|
|
Post #ASoTdoH1CYEa5ilS7s by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@R4_Unit @rich it's less about perfect approximation (that would not be har… |
|
|
|
Post #ASoUXsXsVP6BYbNzzk by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@lritter @rich yeah I’m wondering if it changes what the difficult parts are. |
|
|
|
Post #ASoUXscUEGmjmtXgB6 by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@R4_Unit @rich it will require changes that i can not apply to high dimensional… |
|
|
|
Post #ASoVAch9N3k3gwCJvc by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@lritter @rich activation functions with trainable parameters are not uncommon … |
|
|
|
Post #ASoVAcll5vQbvEM06y by [email protected] |
|
0 likes, 0 repeats |
|
|
|
@R4_Unit @rich yeah but in the end it's supposed to be a ReLU model, so the… |
