Early and accurate detection of pigeon pea leaf diseases is essential for improving crop productivity and ensuring food security, particularly under real-field agricultural conditions. This paper introduces a shallow and computationally off-the-shelf deep learning system to detect the presence of pigeon pea leaf disease with great accuracy and in real-time on resource-limited cameras. DSLR and smartphone cameras were used to make up a custom high-resolution dataset under natural field conditions, including healthy leaves and major diseases, such as Fusarium wilt, leaf spot, and powdery mildew. All the images were downsampled to 224 × 224 pixels and processed with a Gaussian smoothing filter to remove noise and a Canny edge detector to improve structural features. Disease regions were accurately isolated using a Skill Optimization Algorithm (SOA)-driven segmentation strategy that dynamically optimized threshold levels, morphological kernel sizes, and lesion area constraints to handle background clutter and illumination variations. A pretrained EfficientNet-B0 model was used to extract deep semantic features, which consisted of compact 1280-dimensional feature vectors. A novel FMDDCN approach was used to classify these features through exploiting the sensitivity to subtle disease patterns by relying on differential feature modeling and multi-layer fusion of features. The model was fitted on stochastic gradient descent with a learning rate of 1 x 10-3 and a batch size of 32, and assessed on a 60/20/20 train validation test split with 5-fold cross-validation. The results of the experiment show consistent convergence with low overfitting. The proposed framework was found to produce a classification accuracy of 94.5%, precision of 91.0%, recall of 85.5% and Matthews Correlation Coefficient of 88.5% when it was used with four optimized features. In comparison, it is demonstrated that FMDDCN performs better than traditional machine learning and deep learning models, with its F1-score of 0.965 and the overall accuracy of 0.965. The suitability of the real-time edge deployment is verified, as confirmed by the use of computational analysis to reduce inference latency and memory consumption.