Computational methods in data analytics using sparse signal models and learning in high dimensional data with tractable dynamics

Abstract:

We are experiencing a data-driven revolution at the moment with data being collected at an unprecedented rate. In particular, there is an increasing excitement toward systems with learning capabilities. Several data-driven applications have already shown practical benefit of having access to more data. However, large acceptance of such systems heavily depends on stability, tractability and reproducibility features, where current systems fall inadequate to deliver. The scale and complexity of these modern datasets often render classical data processing techniques infeasible, and therefore, several new algorithms in signal processing (SP), optimization, and machine learning (ML) are required to address new technical challenges associated with the nature of the data. In this talk, I will present computational methods in data analytics using sparse signal models and my recent work on analyzing the exact dynamics of iterative algorithms in nonconvex setting that can address aforementioned challenges with examples from low level SP to non-convex ML problems. In the first part of my talk, I will focus on novel computational methods for sampling and reconstruction of signals and images using sparse signal models. In particular, foundations of SP are heavily based on powerful sampling theorems for acquisition, representation and processing. However, increasing evidence shows that the requirements imposed by sampling theorems are too conservative for many naturally-occurring signals, which can be accurately characterized by sparse representations that allows to operate at lower sampling rates closer to the signal’s intrinsic information rates. Finite rate of innovation (FRI) is a new theory that allows to extract underlying sparse signal representations while operating at a reduced sampling rate. In this part, I will present data-driven reconstruction techniques for FRI signals from both theoretical and practical points of view. Specifically, the presentation will cover applications that involve temporal and spatial localization of events in neuroimaging, and in nonlinear tomography for radiating fields. In the second part of my talk, I will present my recent work on analyzing, in the high-dimensional limit, the exact dynamics of iterative algorithms for solving non-convex optimization problems that arise in signal estimation. More specifically, for solving a broad range of learning problems in SP and ML, non-convex optimization problems receive increased interest due to their favorable computational and statistical efficiency. Fortunately, there exists broad range of iterative algorithms (e.g., stochastic-gradient descent (SGD)) with great empirical success. However, there is a lack of true characterization of the dynamics of these methods. Here, I will show how the very high-dimensional settings allow one to apply powerful asymptotic methods to obtain precise characterization. For concreteness, I will focus on the prototypical problem of principal component analysis in an online and distributed setting. I will show that the time-varying empirical measures of the estimates given by the algorithms converge weakly to a deterministic limiting process in the high-dimensional limit. Moreover, this limiting process can be characterized as the unique solution of a nonlinear ODE, and it provides exact information regarding the asymptotic performance of the algorithms. For example, performance metrics such as the MSE, and the cosine similarity can be obtained by examining the deterministic limiting process. A steady-state analysis of the ODE also reveals interesting phase transition phenomena related to the performance of the algorithm. I will conclude my talk by briefly presenting the application of similar analysis techniques to other nonconvex optimization problems.

Bio:

Dr. Dogan is currently a postdoctoral research associate at the John. A. Paulson School of Engineering and Applied Sciences at Harvard University. His main research interests are at the intersection of data analytics, optimization, inverse problems, and machine learning. Apart from the theoretical aspects, his research goals bear practical features in sensor networks, emerging imaging modalities, mining in high dimensional data, and applications of neural networks in imaging and inverse problems. He received his PhD and MSc degrees in electrical engineering from EPFL, Switzerland, in 2015 and 2011, respectively, and his BSc degree in electrical and electronics engineering from METU, Turkey, in 2009.