Blog Archives

Integrating the new ModemManager in GNOME3

(C) allenran917@ Flickr – CC by-nc-nd

It was something like September 2011, shortly after the last Desktop Summit in Berlin, when I started the work improving ModemManager with the new DBus API, GDBus-based DBus support, port-type-agnostic implementations, dynamic DBus interfaces, built-in org.freedesktop.DBus.ObjectManager interface support, built-in org.freedesktop.DBus.Properties.PropertiesChanged signal support, the new libmm-glib client library, the new mmcli command line tool… Took me around half a year to port most of the core stuff, and some more months to port all the plugins we had. And then, suddenly, git master wasn’t that unfinished thing any more, and we were even implementing lots of new features like GPS support in the Location interface, improved SMS Messaging capabilities, ability to switch Firmware on supported modems, and last but definitely not least, support for new QMI-enabled modems through the just officially released libqmi

And when I thought it was all done already, I woke up from my dream and realized that not even me was really using the new ModemManager as it wasn’t integrated in the desktop… not even in NetworkManager. So there I was again, with a whole new set of things to fix…

NetworkManager integration

There is already a set of patches available to include integration of the new ModemManager interface in NetworkManager; pending for review in the NM mailing list. This is the second iteration already, after having some of the patches from the first iteration already merged in git master.

This integration is probably the most complex one in the list, as it really has to deal with quite different DBus interfaces, but the overall look of it seems to be quite good from my point of view. With the patches on, NetworkManager exposes a new –with-modem-manager-1 configure switch which defaults to ‘auto’ (compile the MM1 support if libmm-glib found). Note that both the old and new ModemManager would be supported in this case, to easier fallback to the old version if the user experiences problems with the new ModemManager.

All in all, the required changes to NetworkManager are quite well defined, implemented mainly as a new ‘NMModemBroadband’ object and some additional bits to monitor added/removed modems through the org.freedesktop.DBus.ObjectManager interface.

gnome-shell integration

A proper integration in my desktop of choice (gnome-shell based GNOME 3) required to integrate the new interface in the Network indicator of the shell. As in every good day, I first had to deal with several issues I found, but I ended up submitting a patch that properly talks to the new ModemManager1 interface when needed. As with NetworkManager, this patch would support both the old and the new interfaces at the same time.

Not that this was a big change, anyway. The Network indicator only uses the ModemManager interface to grab Operator Name, MCCMNC and/or SID, and signal strength. Something that could even be included directly in the NMDevice exposed by NetworkManager actually…

Right now the only big issue pending here is to properly unlock the modem when it gets enabled, something that I still need to check how to do it or where to do it.

Truth be told, hacking the shell has ended up being quite nice, even with my total lack of knowledge in JavaScript. My only suggestion here would be: get a new enough distribution with a recent enough GNOME3 (e.g. the unreleased Fedora 18) and jhbuild buildone from there. I think I’ve never seen a clean jhbuild from scratch succeed… maybe it’s just bad luck.

And then came the applet…

network-manager-applet integration

Why the hell would Bluetooth DUN not work with my implementation, I was thinking, until I found that even if GNOME3 doesn’t rely on nm-applet, it still uses some of its functionality through libnm-gtk/libnma, like the code to decide which kind of modem we’re dealing with and launch the appropriate mobile connection wizard. Ended up also submitting patches to include the missing functionality in network-manager-applet.

For those not using GNOME3 and gnome-shell; note that I didn’t implement the support for the new ModemManager1 interface in the nm-applet itself; I just did the bluetooth-guess-my-modem-type bits required in libnma. If I feel in the mood I may even try to implement the proper support in the applet, but I wouldn’t mind some help with this… patches welcome! Same goes for other desktops relying on NetworkManager (KDE, xfce…); I wouldn’t mind to update those myself as well, but I truly don’t have that much free time.

gnome-control-center integration

Ok, so gnome-shell integration is more or less ready now; we should be finished, right? Well, not just yet. gnome-control-center also talks to ModemManager, in this case to get Operator Name and Equipment Identifier, which btw were not getting properly loaded and updated. Once that fixed, I finally got a new patch to have the control center talk to the new interface.

Fedora 18 packages

I bet you won’t go one by one to all the patches I linked before and apply them in your custom compiled NetworkManager, gnome-shell, network-manager-applet and gnome-control-center… but for all those brave Fedora 18 users, you can try with the 64-bit packages that I built for me, all available here:

If you want to rebuild these packages yourself, you’ll find the source tarballs here and the packaging repositories here. The packaging is really awful – I suck at it – so you’ll probably need to install the RPMs with –force :-) Note that the git repo for packaging has several git submodules, one for each item packaged, so remember to “git submodule init” and “git submodule update“.

We done yet?

Are these all the patches needed to have the best ModemManager experience in GNOME3? No! The list of pending tasks for this purpose grows a bit every day…

SMS goodies in ModemManager

(C) josullivan.59 @ Flickr – CC by-nc-sa

Lots of new SMS-related features have landed in ModemManager this week, and as I have as much memory as a fish used to have, I need to write down all the commands I’ve used to test all this; hence this post.

Note that all the stuff I talk about here applies to both AT and QMI based modems.

 

Multiple storages

The current stable 0.5/0.6 releases will only use the core/plugin defined ‘default’ storages for reading and sending short messages. This means that if you happen to have stored SMS in another non-default storage ModemManager won’t show them to you.

In the new 0.7 codebase, ModemManager reads the messages from all the available storages. It is able to read not only DELIVER-type PDUs from received SMS, but also SUBMIT-type PDUs from user-created SMS which got stored. The “Sms.Store()” method in the interface now allows (optionally) specifying in which storage the SMS should be kept.

The following commands allow to create a new SMS and store it in the “sm” (SIM) storage:

$> sudo mmcli -m 0 \
              --messaging-create-sms="text='Hello world',number='+1234567890'

Successfully created new SMS:
    /org/freedesktop/ModemManager1/SMS/21 (unknown)
 
$> sudo mmcli -s 21 --store-in-storage="sm"
successfully stored the SMS
 
$> sudo mmcli -s 21
SMS '/org/freedesktop/ModemManager1/SMS/21'
  -----------------------------------
  Content    |              number: '+1234567890'
             |                text: 'Hello world'
  -----------------------------------
  Properties |            PDU type: 'submit'
             |               state: 'stored'
             |                smsc: 'unknown'
             |            validity: '0'
             |               class: '0'
             |             storage: 'sm'
             |     delivery report: 'not requested'
             |   message reference: '0'

Note that you can create, store and send SMS objects regardless of what storage you’re using for each, ModemManager will take care of properly synchronizing the access to each of them.

The Messaging interface also defines new properties to list all the supported storages and to specify which is the default storage being used for received messages (and for stored ones when no specific one given in “Sms.Store()“):


$> sudo mmcli -m 0 --messaging-status
/org/freedesktop/ModemManager1/Modem/0
  ----------------------------
  Messaging | supported storages: 'sm, me'
            |    default storage: 'me'

 

Text multipart SMS

We support receiving text multipart messages in ModemManager since some stable releases ago; but support for sending multipart messages was still missing. Not any more! Roberto Majadas of OpenShine started this work some months ago with a patch for MM 0.6, which I took over and ported to git master and the new 0.7 codebase.

Of course, we support both the default GSM 7-bit alphabet, as well as UCS-2:

  • GSM 7 encoded multipart messages: If the message is less or equal than 160 septets a singlepart SMS will be used; otherwise the input text will be split into chunks of 153 septets and sent as a multipart message.
  • UCS-2 encoded multipart messages: The GSM 7 alphabet and its extensions are not enough to cover all scripts available out there. Luckily, we can send PDUs with UCS-2 encoded text (UCS-2 covers all the basic plane of Unicode, and is equivalent to UTF-16 in that range). When using this encoding, if the input text is less or equal than 70 UCS-2 characters (140 bytes) a singlepart SMS will be used; otherwise the input text will be split into chunks of 67 UCS-2 characters (134 bytes) and sent as a multipart message. Note anyway that this doesn’t mean that we only allow 70-byte long input strings to fit into a singlepart message; as the conversion from the input UTF-8 text to UCS-2 isn’t always increasing the required number of bytes. As an example, we can now send 210 bytes of UTF-8 encoded chinese text into a single SMS with 140-bytes of UCS-2.

Note that from the API user’s point of view, the SMS objects are not considered multipart or singlepart; that is taken as internal information only required for the PDU-based implementation.

Multipart messages can be created by writing a long enough ‘text‘ value passed in the ‘--messaging-create-sms‘ command, as shown before.

 

Raw data multipart SMS

In addition to text short messages we now also properly support raw binary data singlepart and multipart messages (reported as 8bit encoded). This means that both the sender and receiver can agree in any text encoding to be used between them, or even send in one or more SMS whatever raw binary stream you have.

The ‘mmcli’ command line utility was improved with several commands to test this feature. For example, you can now send using just SMS messages whatever file you have in your system, just doing:

$> sudo mmcli -m 0 \
       --messaging-create-sms="number='+1234567890'" \
       --messaging-create-sms-with-data=/path/to/your/file

Successfully created new SMS:
    /org/freedesktop/ModemManager1/SMS/22 (unknown)
 
$> sudo mmcli -s 22 --send
successfully sent the SMS

When the receiver gets all the parts of the message, she can now recover the sent file with another ‘mmcli’ command in her ModemManager setup:

$> sudo mmcli -m 0 \
              --messaging-list-sms

Found 1 SMS messages:
    /org/freedesktop/ModemManager1/SMS/0 (received)
 
$> sudo mmcli -s 0 \
              --create-file-with-data=/path/to/the/output/file

But beware! The file is sent in chunks of 134 bytes, and there is a maximum of 255 parts in a given multipart SMS, so this leaves you with a maximum size of around 33 Kbytes :-) And not only that, if you’re paying e.g. 0.10€ for each SMS sent, you would be paying more than 25€ for those 33 Kbytes. Yes, needless to say, this is therefore the most useless use case ever. But still, works as a proof of concept.

 

Delivery reports

Last and least, you can now also request delivery reports when sending an SMS, by passing the ‘delivery-report-request=yes‘ key/value pair in the ‘--messaging-create-sms‘ command.


$> sudo mmcli -m 0 \
           --messaging-create-sms="text='你好',\
                                   number='+1234567890',\
                                   delivery-report-request=yes"

Successfully created new SMS:
  /org/freedesktop/ModemManager1/SMS/0 (unknown)
 
$> sudo mmcli -s 0 --send
successfully sent the SMS
 
$> sudo mmcli -m 0 --messaging-list-sms
Found 21 SMS messages:
  /org/freedesktop/ModemManager1/SMS/0 (sent)
  /org/freedesktop/ModemManager1/SMS/1 (received)
 
$> sudo mmcli -s 1
SMS '/org/freedesktop/ModemManager1/SMS/1'
  -----------------------------------
  Content    |              number: '+1234567890'
  -----------------------------------
  Properties |            PDU type: 'status-report'
             |               state: 'received'
             |                smsc: '+9876543210'
             |            validity: '0'
             |               class: '0'
             |             storage: 'unknown'
             |   message reference: '137'
             |           timestamp: '120914120619+02'
             |      delivery state: 'temporary-error-sc-specific-reason' (0x30)
             | discharge timestamp: '120914120619+02'

You can then match the delivery report with the message sent by using the ‘message reference‘ value, which should be the same in both. This also applies to multipart messages, where the ‘message reference’ of our SMS object is the one reported when sending the last part, and therefore we only request the delivery report in the last part sent.

Enjoy!

An introduction to libqmi

What is QMI?

QMI is a binary protocol designed to replace the AT command based communication with modems, and is available in devices with Qualcomm chipsets from multiple vendors (Novatel, Huawei, Sierra Wireless, ZTE… and of course Qualcomm itself).

The protocol defines different ‘services‘, each of them related to different actions that may be requested to the modem. For example, the ‘DMS’ (Device Management) service provides actions to load device information; while the ‘NAS’ (Network Access) service provides actions to register in the network. Similarly, other services will allow the user to request data connections (WDS), setup GPS location reporting (PDS), or manage internals of the user identity module (UIM service). The user needs to handle the creation of ‘clients’ for those services by allocating/deallocating ‘client IDs’ using the generic always-on ‘control’ (CTL) service.

Each service in the protocol defines ‘Request and Responses‘ as well as ‘Indications‘. Each pair of request/response has a maching ID which lets the user concatenate multiple requests and get out-of-order responses that can be afterwards matched through the common ID. Indications arrive as unsolicited messages, sent either to a specific client or as a broadcast message to all the clients of a given service. Usually the user needs to request the enabling of the indications to receive via some request/response.

Finally, each message in the protocol defines a set of input (in requests) and output (in responses and indications) arguments, which we name as TLVs. Some of these are defined to be mandatory, others are optional, and others are only mandatory if some other argument is available and has a given value. For example, some output arguments in response messages are only mandatory if the result of the response (given as a TLV) is SUCCESS.

 

Using the QMI protocol

This protocol is easily accessible in recent enough Linux kernels (>= 3.4), through the cdc-wdm and qmi_wwan drivers. Once these drivers are in place and the modem gets plugged in, the kernel will expose a new /dev/cdc-wdm device which can talk QMI, along with a wwan interface associated to each QMI port.

 

libqmi

libqmi is an on-going effort to get a library providing easy access to Qualcomm’s ‘QMI’ protocol. The current libqmi available is based on the GLib/GObject/GIO-powered ‘libqmi-glib’. libqmi tries to ease the use of the protocol by providing:

  • A ‘QmiDevice‘ object to control the access to the /dev/cdc-wdm device. This object allows creating new per-service ‘QmiClient’s. It also hides the use of the implicit ‘CTL’ service client.
  • The ‘QmiClient‘ object provides an interface to the set of request/responses of a given service. Users of the library will just need to execute a GIO-asynchronous method to send the request and get the processed response.
  • The ‘QmiClient‘ object also provides signals for each of the indications given in the service.
  • The input and output arguments needed in requests, responses and indications are compiled into ‘bundles’. These bundles are opaque structs with accessors for their elements, which allow us to add new TLVs to a given message in the future without breaking API/ABI.

The protocol is written in the library as a JSON dictionary of rules, and most of the code to handle the protocol in the library is autogenerated during compilation. If you want to take a look at the currently available interface, check the gtk-doc documentation at: http://www.freedesktop.org/software/libqmi/libqmi-glib/latest/

 

qmicli

The libqmi project comes with a command line utility (qmicli) primarily used as a developer tool for testing the libqmi-glib library. Just run the program with ‘--help-all‘ to get all possible actions it can run. A quick example of usage:

$> sudo qmicli -d /dev/cdc-wdm0 --dms-get-manufacturer
[/dev/cdc-wdm0] Device manufacturer retrieved:
Manufacturer: 'Qualcomm Incorporated'

Hint: If you compile the project passing -DMESSAGE_ENABLE_TRACE to CFLAGS, it will dump as DEBUG logs the content and translation of all the QMI messages passed in both directions. You’ll also need ‘--verbose‘ or just ‘-v‘ to show the debug logs when running qmicli.

 

Other utilities

The project also comes with a small bash script called ‘qmi-network‘, which allows you to launch a broadband connection using libqmi-glib and qmicli. This script requires two arguments; first the cdc-wdm device to talk to, and secondly the action (start|stop|status) to execute. The script can definitely be improved in multiple ways (e.g. send PIN), so if anyone wants to do it patches are very welcome.

 

Where do I get it?

The project is currently available in the freedesktop.org infrastructure:

 

ModemManager

ModemManager has QMI support through libqmi-glib since release 1.0.

Help! Contribute! Sponsor!

If you want to help, or sponsor further development in libqmi, qmicli or the ModemManager integration, just let me know!

[UPDATE] Links updated, along with ModemManager related info.

Follow

Get every new post delivered to your Inbox.

Join 36 other followers