Brain disorders are often linked to disruptions in the dynamics of the brain's intrinsic functional networks. It is crucial to identify these networks and determine disruptions in their interactions to classify, understand, and possibly cure brain disorders. Brain's network interactions are commonly assessed via functional (network)\ connectivity, captured as an undirected matrix of Pearson correlation coefficients. Functional connectivity can represent static and dynamic relations. However, often these are modeled using a fixed choice for the data window. Alternatively, deep learning models may flexibly learn various representations from the same data based on the model architecture and the training task. The representations produced by deep learning models are often difficult to interpret and require additional posthoc methods, e.g., saliency maps. Also, deep learning models typically require many input samples to learn features and perform the downstream task well. This dissertation introduces deep learning architectures that work on functional MRI data to estimate disorder-specific brain network connectivity and provide high classification accuracy in discriminating controls and patients. To handle the relatively low number of labeled subjects in the field of neuroimaging, this research proposes deep learning architectures that leverage self-supervised pre-training to increase downstream classification. To increase the interpretability and avoid using a posthoc method, deep learning architectures are proposed that expose a directed graph layer representing the model's learning about relevant brain connectivity. The proposed models estimate task-specific directed connectivity matrices for each subject using the same data but training different models on their own discriminative tasks. The proposed architectures are tested with multiple neuroimaging datasets to discriminate controls and patients with schizophrenia, autism, and dementia, as well as age and gender prediction. The proposed approach reveals that differences in connectivity among sensorimotor networks relative to default-mode networks are an essential indicator of dementia and gender. Dysconnectivity between networks, especially sensorimotor and visual, is linked with schizophrenic patients. However, schizophrenic patients show increased intra-network default-mode connectivity compared to healthy controls. Sensorimotor connectivity is vital for both dementia and schizophrenia prediction, but the differences are in inter and intra-network connectivity.