Non-invasive assessment of retinal fundus image is well suited for early detection of ocular disease and is facilitated more by advancements in computed vision and machine learning. Most of the Deep learning based diagnosis system gives just a diagnosis(absence or presence) of a certain number of diseases without hinting the underlying pathological abnormalities. We attempt to extract such pathological markers, as an ophthalmologist would do, in this thesis and pave a way for explainable diagnosis/assistance task. Such abnormalities can be present in various regions of a fundus image including vasculature, Optic Nerve Disc/Cup, or even in non-vascular region. This thesis consist of series of novel techniques starting from robust retinal vessel segmentation, complete vascular topology extraction, and better ArteryVein classification. Finally, we compute two of the most important vascular anomalies-arteryvein ratio and vessel tortuosity. While most of the research focuses on vessel segmentation, and artery-vein classification, we have successfully advanced this line of research one step further. We believe it can be a very valuable framework for future researcher working on automated retinal disease diagnosis.