NeurIPS 2019杰出论文深度解读:窥视机器学习的核心问题
共 4529字,需浏览 10分钟
· 2019-12-24
新智元推荐
来源:学术头条(ID:SciTouTiao)
作者:AMiner编辑部
【新智元导读】在NeurIPS 2019一千多篇入选论文中,有那么1篇杰出论文值得长时间、深入、反复学习。这篇杰出论文的出彩之处在哪里呢?我们又能从中学到什么呢?现在戳右边链接上新智元小程序 了解更多!
该论文的作者是:Ilias Diakonikolas(威斯康辛大学麦迪逊分校)、Themis Gouleakis(马克斯普朗克计算机科学研究所)、Christos Tzamos(威斯康辛大学麦迪逊分校)。
论文地址:https://papers.nips.cc/paper/8722-distribution-independent-pac-learning-of- halfspaces-with-massart-noise
前言
本文将对NeurIPS 2019杰出论文《Distribution-Independent PAC Learning of Halfspaces with Massart Noise》进行解读,该论文在半空间学习上取得了显著进展。作者研究了存在Massart噪声的半空间(halfspaces)分布独立的PAC学习问题。具体而言,给定从R^d+1上的分布D中提取的一组有标签样本(x, y),使无标记点x上的边缘分布是任意的,且标签y由未知半空间生成,该空间被噪声率为η<1/2的Massart噪声破坏。最终目标是找到一个分类器h,使错误分类误差最小。对于具有错误分类误差η+ε的问题,作者给出了一个poly(d, 1/ε)时间算法。作者还证明了对算法的误差保证进行改进可能很难实现。在此工作之前,即使是析取类,在这个模型中也没有有效的弱学习方法(分布独立)。
研究现状
Massart 噪声与RCN
随机分类噪声(Random Classification Noise ,RCN)【1】是Massart噪声的特殊情况,其每个标签的翻转概率恰好为η<1/2。似乎Massart噪声比RCN更易于处理。但实际上,Massart对抗需要选择是否扰动给定的标签,如扰动,以何种概率进行,因此,在该模型中设计有效的算法具有很大挑战性。尤其是,RCN学习与统计查询(Statistical Query,SQ)模型【2】【3】之间的联系不再成立,即,作为SQ算法的性质已不能自动满足用Massart噪声进行噪声容忍学习(noise-tolerant learning)的需要。而【4】【5】中正是利用了RCN与SQ模型的关系,得到了用RCN学习半空间的多项式时间算法。
相关工作介绍
![16b01c4db7ae307d191fceee1d6dd7f3.webp](https://filescdn.proginn.com/f8c5a85fdbc2d22c6ca812869a7095f8/16b01c4db7ae307d191fceee1d6dd7f3.webp)
方法
相关基础
![0999161516947674a558489a9b28ccfd.webp](https://filescdn.proginn.com/61493aa0a8a8088790cb84d48f917ac1/0999161516947674a558489a9b28ccfd.webp)
带Massart噪声的半空间学习算法
![2849c4f3568e74645cb7be8a928be6df.webp](https://filescdn.proginn.com/7783c4c87991cb59a16a3d0739730745/2849c4f3568e74645cb7be8a928be6df.webp)
![41c62906cce709b9098a2b5950cab16f.webp](https://filescdn.proginn.com/8d4a707489747fa6aea5e5c884319c28/41c62906cce709b9098a2b5950cab16f.webp)
学习大边界半空间
![a7f4e5ca1e1ba6983fa01f287ffb3780.webp](https://filescdn.proginn.com/a89db7d00d6beee98ec60f13de84185e/a7f4e5ca1e1ba6983fa01f287ffb3780.webp)
![978ebea65adb24cfb8535c84c3f0fd8b.webp](https://filescdn.proginn.com/368aa6b2b0a5f039dd4b2dd277da3417/978ebea65adb24cfb8535c84c3f0fd8b.webp)
![65584b15053f87f5eed9d1a9ff444ebf.webp](https://filescdn.proginn.com/8f794a39cf9a5bf1e031ec1ec11fbbe0/65584b15053f87f5eed9d1a9ff444ebf.webp)
![dd3903affffa75530d920608c3fe0f6e.webp](https://filescdn.proginn.com/bb00e41bd5877543f2f2ad4b8d3b442b/dd3903affffa75530d920608c3fe0f6e.webp)
![1e488d30e5aa65ce85231d973c323c6a.webp](https://filescdn.proginn.com/4d1c70180ab9042e0dc93b0d132409f3/1e488d30e5aa65ce85231d973c323c6a.webp)
一般情况
![370b76a588321ce1c44cca3da4f7e2aa.webp](https://filescdn.proginn.com/e2d1a663a6e18268267e75135af561fa/370b76a588321ce1c44cca3da4f7e2aa.webp)
![61703bb4bcc14515e66ce9abdb8b4ca4.webp](https://filescdn.proginn.com/b2b5180c69e218a13640b1364d0c8a1e/61703bb4bcc14515e66ce9abdb8b4ca4.webp)
![2e359357638bba4c242e54f4289c7d86.webp](https://filescdn.proginn.com/fb5f9b02beea8d1cd14bafc5a0ae4191/2e359357638bba4c242e54f4289c7d86.webp)
令D为(d + 1)维度的带标签样本在b-bit复杂度上的分布,由一个未知的半空间所产生,该空间被噪声率为η<1/2的Massart噪声破坏。算法2使用个样本,运行时间为poly(d, 1/ε, b),最终以2/3的概率返回一个分类器h,且其误分类误差
。
【1】D. Angluin and P. Laird. Learning from noisy examples. Mach. Learn., 2(4):343–370,1988.【2】M. J. Kearns. Efficient noise-tolerant learning from statistical queries. In Proceedings of the Twenty-Fifth Annual ACM Symposium on Theory of Computing, pages 392–401,1993.【3】M. J. Kearns. Efficient noise-tolerant learning from statistical queries. Journal of the ACM, 45(6):983–1006, 1998.【4】A. Blum, A. M. Frieze, R. Kannan, and S. Vempala. A polynomial-time algorithm for learning noisy linear threshold functions. In 37th Annual Symposium on Foundations of Computer Science, FOCS ’96, pages 330–338, 1996.【5】A. Blum, A. Frieze, R. Kannan, and S. Vempala. A polynomial time algorithm for learning noisy linear threshold functions. Algorithmica, 22(1/2):35–52, 1997.【6】T. Bylander. Learning linear threshold functions in the presence of classification noise.In Proceedings of the Seventh Annual ACM Conference on Computational Learning Theory, COLT 1994, pages 340–347, 1994.【7】A. Blum, A. M. Frieze, R. Kannan, and S. Vempala. A polynomial-time algorithm for learning noisy linear threshold functions. In 37th Annual Symposium on Foundations of Computer Science, FOCS ’96, pages 330–338, 1996.【8】E. Cohen. Learning noisy perceptrons by a perceptron in polynomial time. In Proceedings of the Thirty-Eighth Symposium on Foundations of Computer Science, pages 514–521,1997.【9】J. Dunagan and S. Vempala. A simple polynomial-time rescaling algorithm for solving linear programs. In Proceedings of the 36th Annual ACM Symposium on Theory of Computing, pages 315–320, 2004【10】P. Awasthi, M. F. Balcan, N. Haghtalab, and R. Urner. Efficient learning of linear separators under bounded noise. In Proceedings of The 28th Conference on Learning Theory, COLT 2015, pages 167–190, 2015.【11】Y. Zhang, P. Liang, and M. Charikar. A hitting time analysis of stochastic gradient langevin dynamics. In Proceedings of the 30th Conference on Learning Theory, COLT 2017, pages 1980–2022, 2017.【12】Y. Zhang, P. Liang, and M. Charikar. A hitting time analysis of stochastic gradient langevin dynamics. In Proceedings of the 30th Conference on Learning Theory, COLT 2017, pages 1980–2022, 2017.【13】P. Awasthi, M. F. Balcan, N. Haghtalab, and H. Zhang. Learning and 1-bit compressed sensing under asymmetric noise. In Proceedings of the 29th Conference on Learning Theory, COLT 2016, pages 152–192, 2016.【14】A. Dvoretzky, J. Kiefer, and J. Wolfowitz. Asymptotic minimax character of the sample distribution function and of the classical multinomial estimator. Ann. Mathematical Statistics, 27(3):642–669, 1956.【15】J. Dunagan and S. Vempala. Optimal outlier removal in high-dimensional spaces. J.Computer & System Sciences, 68(2):335–373, 2004.
本文经授权转载自:学术头条
![820e1705bfe993471d32c5ed527a96d8.webp](https://filescdn.proginn.com/0ca64b0ca07d1a5559d05734b68a8c5f/820e1705bfe993471d32c5ed527a96d8.webp)