Evolution Of Activation Functions for Neural Architecture Search

الموضوعات

علوم تطبيقية وتكنولوجية

الملخص

The introduction of the ReLU function in neural network architectures yielded substantial improvements over sigmoidal activation functions and allowed for the training of deep networks. Ever since, the search for new activation functions in neural networks has been an active researchtopic. However, to the best of our knowledge, the design of new activation functions has mostly been done by hand. In this work, we propose the use of a self-adaptive evolutionary algorithm that searches for new activation functions using a genetic programming approach, and we compare the performance of the obtained activation functions to ReLU. We also analyze the shape of the obtained activations to see if they have any common traits such as monotonicity or piece-wise linearity, and we study the effects of the self-adaptation to see which operatorsperform well in the context of a search for new activation functions. We perform a thoroughexperimental study on datasets of different sizes and types, using different types of neural network architectures. We report favorable results obtained from the mean and standard deviation of the performance metrics over multiple runs.

روابط وملفات

التعريف والنوع

رقم الوثيقة

0b8900dd-44a3-4987-b278-e98c2d5f3b19

رقم العقد

0

نوع الوسائط

Crawler

نوع المحتوى

الرسائل العلمية

صيغة المصدر

رسائل ماجيستير

نوع الملف

pdf text

أسماء الملفات

2212726_1.pdf

بيانات النشر

ألقاب المؤلفين

[{"name_ar":"Andrew Nader","title_ar":"اعداد","title_en":"Preparation"},{"name_ar":"Danielle Azar ","title_ar":"اشراف","title_en":"Supervision"}]

اللغة

English

المصدر والدورية

اسم المصدر

Evolution Of Activation Functions for Neural Architecture Search

المحتوى والصفحات

عدد الصفحات

0

كلمات الباحثين

Functions for Neural

إشراف وإعداد

الإشراف

Danielle Azar

الإعداد

Andrew Nader

الاقتباسات الببليوغرافية