We are using Moodle 3.4 on one of our platforms and we have messaging turned off. Because of the way the messaging and notifications are built, this means that no notifications are being sent as new notification to our users.

As far as I found out if the messaging is turned off ( $CFG->messaging  = 0 ), whenever a new notification is created ( eg. about a new assignment [ this is good example as this is an instant notification, you'll get it as soon as you submit your Assignment]), it is then marked as read in the same second.

Although we use 3.4, I was able to reproduce the bug on the demo.moodle.net which is Moodle 3.5.2 (Build: 20180910).

Steps to reproduce

// notice: if you test this on the demo.moodle.net site, make sure there is atleat 15min left until clock goes next hour as the site is reset every 60 minutes //

Login as admin/sandbox on demo.moodle.net and add 2 assignments to Course 1, call them 'Messaging ON' and 'Messaging OFF' for clearence. Log out.

Login with student/sandbox. Make sure you click the notifications bell and turn on notifications when online for WEB for new Assignments  or CTRL + click [here|https://demo.moodle.net/message/notificationpreferences.php?userid=4] 

Submit 'Messaging On' assignment in Course 1 and observe your new notification in the right top corner. Don't click on the notification just yet. Log out.

Login with admin/sandbox and turn off messaging from Site Administration/Advanced features or CTRL + click here and tick it off

Login with student/sandbox and submit 'Messaging OFF' assignment and observe that you don't have any new notifications ( apart from the one you had from submitting the first Assignment ).

Click the notifications bell and observe that your assignment for Messaging OFF has been marked as read. 

Open up developer tools/Network, refresh the page and click notifications. After this open up service.php?sesskey=seSSkeyExample on right side and observe that the first notification object in the data(notifications) array is marked as read as soon as its created or see it on the screenshot below

link to the screenshot: https://ibb.co/eiTRRe 

A possible solution to this it would be to place a $CFG->notifications = 0 in the site config. And add admin settings just under the Enable messaging system. If then we turn off messaging but we want our users to receive notifications, we can set Notifications to allow(1). And when /lib/classes/message/manager.php :88 and/or :159 would do their checks to mark all messages as read, we could implement/update these conditions to 

/lib/classes/message/manager.php:88

if ($savemessage->notification or empty($CFG->messaging) && (empty($CFG->notifications))) {

    // If they have deselected all processors and its a notification mark it read. The user doesn't want to be bothered.

    // The same goes if the messaging is completely disabled.

    // We cannot insert directly to the message_read table because we want to get all events in proper order!

    $messageid = message_mark_message_read($savemessage, time(), true);

} else {

    // Just add it to the list of unread messages, there is no way it could be delivered to them,

    // but they can read it via the messaging UI later.

    $messageid = $savemessage->id;

}

return $messageid;

/lib/classes/message/manager.php:161

if (empty($CFG->messaging) && empty($CFG->notifications)) {

    // If messaging is disabled and they previously had forum notifications handled by the popup processor

    // or any processor that puts a row in message_working then the notification will remain forever

    // unread. To prevent this mark the message read if messaging is disabled.

    $messageid = message_mark_message_read($savemessage, time());

} else if ($failed) {

    // Something failed, better keep it as unread then.

    $messageid = $savemessage->id;

} else if ($DB->count_records('message_working', array('unreadmessageid' => $savemessage->id)) == 0) {

    // If there is no more processors that want to process this we can move message to message_read.

    $messageid = message_mark_message_read($savemessage, time(), true);

} else {

    // Some processor is still working on the data, let's keep it unread.

    $messageid = $savemessage->id;

}

return $messageid;

add advanced feature admin setting:

/admin/settings/subsystems.php:18

$optionalsubsystems->add(new admin_setting_configcheckbox('notifications', new lang_string('notifications', 'admin'), new lang_string('confignotifications', 'admin'), 1));

add langauge strings:

/lang/en/admin.php:272

$string['confignotifications'] = 'Should notifications be enabled ?';

I've tested and if I add these changes and turn messaging off, notifications on,  users can see notifications as new notifications. Also If I turn notifications off as well, it will behave as it was before.