Development of self-driving cars aims to drive safely from one point to another in a coordinated system where the on-board autonomous vehicle system should react and possibly alert drivers about the driving environments and possible collisions that may arise between drivers and obstacles. In order to achieve a high level of autonomy in urban scenarios with unpredictable traffic, these systems must have robust and reliable obstacles detection systems. This work proposes U19-Net, a deep learning model that explores improvement of the so-called encoder-decoder neural networks with a very deep network architecture. In particular, U19-Net is applied and successfully evaluated for the vehicle and pedestrian detection tasks within an open source dataset consisting of frames from a video in real driving scenarios. The output of the network consists of a pixel-level mask that identifies each pixel as vehicle or pedestrian, demonstrating that the depth representation attained with U19-Net is beneficial in this kind of architecture for vision systems in self-driving cars.