November 17, 2007

Lego NXT + wiimote with MSRS tutorial

It's time to post the first tutorial on this blog: controlling a Lego Mindstorms NXT robot (but the code would also work for any other differential drive robot supported by MSRS, including the BASIC Stamp-based BOE-bot from Parallax, the iRobot Create, etc.) with a Nintendo wiimote (if you're not yet an expert: the WII controller) using Microsoft Robotics Studio (MSRS) 1.5. I wrote an introduction to this in my first post, and the time has come to write a tutorial about it! I'm also going to introduce you to writing services with Microsoft Robotics Studio.

Update: You can download a project for Microsoft Robotics Developer Studio 2008 here (read the readme first). Most of the tutorial is still compatible with the new version, so there shouldn’t be any problems following it.

Creating a service

A service is a fundamental concept in MSRS. A service can represent many different things in MSRS:

  1. First, every application you write with MSRS is a service by itself
  2. The code which allows communication with a robot is a service. It’s sometimes called ‘BrickService’ and it sends motor data/receives sensor data using the appropriate communication protocol (e.g. the PC communicates with the NXT via Bluetooth and sends/gets data in a specific way)
  3. The ‘generic contracts’ are services that allow you to handle specific parts of a robot, like motors, a differential drive system, different sensors, cameras. These services are the same for whatever robot you are using. That allows you to run the exact same program with different robots (for which the contracts you’re using are available) or even in simulation.
  4. You can even write a service for a specific behavior, so that you could use that service in another application without having to write the whole thing again. (a good example would be the Robotics Tutorial 3 - Creating Reusable Orchestration Services on msdn)
  5. Any other piece of code for MSRS…

A service is always communicating with other services: sending requests (e.g. telling the drive service to go forward at 50% speed for the left and right motors), and receiving notifications from another service to which you have subscribed (for example, a bumper service would send a notification when the bumper was pressed or released, or the wiimote service would send notifications when the state is changed, i.e. when the accelerometer or a button changes state).

 

So after this brief intro about the structure of an application, let’s start by creating a service!

To do this, you need to open the MSRS command prompt (this is an important tool in MSRS), found by navigating from the start menu. It’s nothing more than a console window waiting for you to write commands. The command you need to create a service is dssnewservice, followed by a set of parameters. The only parameter you really need for now is /service: (or just /s:), where you specify the name of the service you want to create (others include the programming language, the namespace, etc.). So let's create a wiimoteNxt service:

dssnewservice /s:wiimoteNxt

This creates a folder 'wiimoteNxt' in the C:\Microsoft Robotics Studio (1.5)\ directory (or wherever you installed MSRS 1.5). Navigate to it, and open the wiimoteNxt.sln solution file with Visual C# express or any other version of Visual C#/Visual Studio you have. And there's the project of you service!

Adding references

The next thing we need to do is to add references to our project, so that we can use what we need to use: a drive service, and Brain Peek's WiimoteLib service.

 

Update: For the next part, note that I provided the necessary wiimoteLib files with the MRDS 2008 version of the project, so you won’t need to migrate the project, although downloading wiimoteLib can be useful for additional stuff.

First, let's set up the WiimoteLib service, so that we can use it in our project. If you haven't downloaded it yet, you can get it here (get the version with the source code). After extracting, you can see a few folders. The folder for the MSRS service is WiimoteCS\WiimoteMSRS, but the problem is that it was written for MSRS 1.0, not 1.5 (Update: the new versions of wiimoteLib are written for MSRS 1.5, so you shouldn’t need to migrate the project, but you still do need to build it). Fortunately, MSRS 1.5 has a nice tool to convert existing services prior to 1.5 to 1.5: the dssProjectMigration tool. To convert the service, just type the following command in the Console, with the appropriate path:

dssProjectMigration "D:\wiimote\291133_WiimoteLib\WiimoteCS\WiimoteMSRS"

Once you've done this, you can compile the service, which is now 1.5 compatible. To do this, open the Wiimote.sln file, then Build->Build Wiimote, then close the solution.

 

Now that we have compiled the WiimoteLib service, we are going to be able to use it in our wiimoteNxt service. Now we have to add the references: right click on 'References' in the Solution Explorer, and click 'Add Reference...', as shown:

 

Capture

 

In the window that pops up, select the "Wiimote.Y2007.M06.Proxy" reference, which corresponds to the service you've just built, in the .NET tab (every time you'll want to add a reference, you'll have to choose the component ending with .Proxy; the others have other purposes):

 

Capture

 

Also add the RoboticsCommon.Proxy reference. That's where the drive service is.

Next, we'll have to add using directives in our main .cs file, so that we won't need to type Microsoft.Robotics.Services.Drive.Proxy.something, but only drive.something. Add the following code at the top of WiimoteNxt.cs, after the other using directives:

using wiimote = WiimoteLib.Proxy;
using drive = Microsoft.Robotics.Services.Drive.Proxy;

Now we're all set! We can finally start to write something interesting!

Subscribing to the wiimote

In order to get notifications from the wiimote (i.e. acceleration data for us), we have to subscribe to its service. This allows the wiimote service to know that it will have to send us notifications, each time a change occurred.

 

When you subscribe to a service, that service will send messages with the notifications, therefore, our service will need something to receive those messages, in a sort of stack that holds those messages. In MSRS, this is called a Port. Every service has its own port, usually defined as "Operations", which gives us access to the service.

 

To specify that the two services are going to communicate, we'll need to add a Partner attribute before declaring the ports (attributes are an advanced feature of C#.NET, which I won't cover, but be aware that it's just a way to tell the runtime environment something about what is running. For instance, MSRS uses that information during runtime to show which services are running, what they subscribe to, etc. on a browser). The wiimote and drive services are called Partner services.

For the wiimote, we will need to declare two ports: one is the main wiimote port, declared with a Partner attribute, and the other is a notification port, which is used to receive notifications and handle them. The parameters of the Partner attribute are:

  1. The name identifying the partner (this name will appear on the web browser, for instance)
  2. The ID of the service (a complicated string belonging to the service), accessed with: serviceProxy.Contract.Identifier
  3. The creation policy of the partner. This tells if a new instance of the service should be created or not (for instance, the wiimote data is the same everywhere, so you could use the same instance in different services; whereas a behavior service is different for different robots, so you should create a new instance in different services). Here we will be using PartnerCreationPolicy.UseExistingOrCreate, which means that if there already is an instance, we'll use that one, otherwise, we'll create a new one.

Now copy this code after the _mainPort declaration, in the same file.

//the wiimote port (with the appropriate Partner attribute)
[Partner("wiimote", Contract = wiimote.Contract.Identifier, CreationPolicy = PartnerCreationPolicy.UseExistingOrCreate)]
private wiimote.WiimoteOperations _wiiPort = new wiimote.WiimoteOperations();

//the wiimote notification port
private wiimote.WiimoteOperations _wiiNotify = new wiimote.WiimoteOperations();

Now that we have got the ports, we need to subscribe to the wiimote service. In order to do that, the main wiimote port has to subscribe to the notification port, so that it will get the notifications from it. Add the following line of code in the Start() function, after base.Start():

_wiiPort.Subscribe(_wiiNotify);

Then, we have to start receiving notifications from the wiimote. That is, you need to activate a receiver, which will receive wiimote.WiimoteChanged messages from the _wiiNotify port, and go on receiving forever. This sentence translates into the following code, which follows the previous line (true stands for 'forever', whereas false would be 'just once'):

Activate(
    Arbiter.Receive<wiimote.WiimoteChanged>(true, _wiiNotify, wiimoteChangedHandler)
);

Notice that the last argument is 'wiimoteChangedHandler'. That's a handler we haven't declared yet, which will be executed each time a new wiimote.WiimoteChanged message comes in, with that message as an argument to the handler. So let's declare that! (you can add it after the GetHandler):

//wiimote notifications handler
void wiimoteChangedHandler(wiimote.WiimoteChanged wiimoteChanged)
{
 
}

We are ready to send requests to the motors!

Sending requests to the motors

Now that we have a handler for the wiimote state changes, we only need to send the appropriate requests to the NXT motors.

First, let's add the using directive:

using drive = Microsoft.Robotics.Services.Drive.Proxy;

Then the Partner drive port:

//The drive port with its Partner attribute
[Partner("drive", Contract = drive.Contract.Identifier, CreationPolicy = PartnerCreationPolicy.UseExistingOrCreate)]
private drive.DriveOperations _drivePort = new drive.DriveOperations();

Now, in the wiimote handler, we'll have to send the drive requests to the motors. You can set up the request with the drive.SetDrivePowerRequest class, and then send it with the _drivePort.SetDrivePower() function. Here is the updated handler, with comments:

void wiimoteChangedHandler(wiimote.WiimoteChanged wiimoteChanged)
{
    //swap some variables to use the wiimote horizontally (float between -1.0f and 1.0f)
    float x = -wiimoteChanged.Body.AccelState.Values.Y;
    float y = wiimoteChanged.Body.AccelState.Values.X;
 
    //create a drive request
    //(if you prefer, there is a constructor that allows you to set the speeds directly):
    //drive.SetDrivePowerRequest request = new drive.SetDrivePowerRequest(leftSpeed, rightSpeed);
    drive.SetDrivePowerRequest request = new drive.SetDrivePowerRequest();
    
    request.LeftWheelPower = y + x;
    request.RightWheelPower = y - x;
 
    //OR (if you feel uncomfortable with the way it's going backwards)
    //if (y > 0)
    //{
    //    request.LeftWheelPower = y + x;
    //    request.RightWheelPower = y - x;
    //}
    //else
    //{
    //    request.LeftWheelPower = y - x;
    //    request.RightWheelPower = y + x;
    //}
 
    _drivePort.SetDrivePower(request); //send the request to the drive port
}

And that's it!

Running the service

Before running the program, we have to specify a manifest file, that tells MSRS which robot to use (in this case, we only need a robot that supports the drive service, which is most of the mobile robot), or even run in simulation.

 

If you want to run the service from within Visual Studio (and debug the program), you have to change the command line arguments in the Properties (either from the Solution Explorer, or Project->Properties), Debug tab.

 

Here are a few manifests to add at the end of the 'Command line arguments' textbox:

  1. Lego NXT robot (Tribot or any other custom differential drive robot): "samples\config\LEGO.NXT.TriBot.manifest.xml"
  2. Tribot in simulation: "samples\config\LEGO.NXT.TriBot.simulation.manifest.xml"
  3. BOE-Bot: "samples\config\Parallax.BoeBot.Drive.manifest.xml"
  4. iRobot: "samples\config\iRobot.manifest.xml"
  5. iRobot Create in simulation: "samples\config\IRobot.Create.Simulation.xml"

Capture 

 

 

DSC00042

Then, after connecting the robot and the wiimote to the PC (I use BlueSoleil on the PC, which works great compared to the original drivers I had. The wiimote didn't work before, now it does with BlueSoleil, so you might want to try it if you have any problems), debug the program (press green arrow button, or F5), and...there you go!!! (hold the wiimote as shown)

 

Another way to run the service without opening Visual Studio is through the MSRS Command Prompt, with the following command:

dsshost -port:50000 -tcpport:50001 -manifest:"wiimoteNxt/wiimoteNxt.manifest.xml" -m:"samples/config/Lego.NXT.Tribot.manifest.xml"

(note that m is short for manifest, and -m: and /m: are the same)

Change the paths of the manifests, if needed (the first one is the service's manifest, and the second is the robot's one), get ready, and have fun!

 

Finally, here's a video:

 

If you need any help with connecting and initializing your robot, or anything else about MSRS, check the MSRS documentation (online, or in chm format in the installation folder).

 

You can download the project here (extract it in the MSRS 1.5 directory)

You can get the MRDS 2008 version here (read the readme.txt first)

33 comments:

tingeypa said...

Nice work. Does the MSRS need to be installed to use the remote/NXT combination - or can it produce an executable? Can you add a userinstace as well?

Regards, Paul

Al said...

MSRS is a framework and needs to be installed in order to run services, kind of like DirectX for 3D graphics.

You can't produce executables for the moment, but you can definitely write a batch file with the appropriate command to run in the console (you'll always need the console), where you would write something like:
bin\dsshost /p:50000 /m:config/yourService.manifest.xml
But even if you had an executably, you would still need to have MSRS installed!

What do you mean by userinstance?
If it's something .NET-related, you can do anything you want, from Windows Forms to networking, and also any kind of library that supports .NET (MSRS is all based on .NET)

Hope this helps,
Alberto

tingeypa said...

Thanks for your reply. Sorry I missed-typed userinstance - I ment User Interface, like a winforms app. I am assuming that since dsshost is required, MSRS is actually a runtime so I can't just call the NXT/wiimote as libraries from a winforms app. In other words, I am not in control of the threads to drive a windows interface. Maybe I could make a seperate interface and communicate with via remoting or sockets?

One more question - do I need to change the NXT firmware to create a bluetooth controlled robot or will it work with the Lego firmware?

Thanks, Paul

Al said...

MSRS services run with dsshost, but all the code is written in .NET, so when you're writing a service, you can of course add a windows forms file to your project in Visual Studio, and make the user interface you need, separately from the msrs service. Then all you need to do is run the UI in a separate thread in the Base function of the dss service, and send/receive messages from and to each thread.

A way of doing that is described in the robotics tutorial 4 of the MSRS documentation. (you can also have a look at the other numerous tutorials they have)

You don't need to change the firmware to run services: the msrs runtime automatically starts a program called msrs(written in NXT-G, which means it uses the lego firmware) on the brick, which sends sensor data and receives motor data from and to the computer, in a loop, so that the computer can use the data, process it and send back actuator data to the robot, via bluetooth in this case.

Alberto

tingeypa said...

Thanks Alberto! I'll give it a try.

Evan said...

i get two build errors and i dont know how to fix it please help if you need a picture ill post it but here is the error list:
Error 1 The OutputPath property is not set for this project. Please check to make sure that you have specified a valid Configuration/Platform combination. Configuration='Debug' Platform='HPD' wiimoteNxt
Error 2 The command ""C:\Microsoft Robotics Studio (1.5)\bin\dssproxy.exe" /dll:"C:\Microsoft Robotics Studio (1.5)\bin\wiimoteNxt.Y2007.M11.dll" /proxyprojectpath:"C:\Microsoft Robotics Studio (1.5)\wiimoteNxt\Proxy " /keyfile:"C:\Microsoft Robotics Studio (1.5)\samples\mrisamples.snk" /binpath:". " /referencepath:"C:\Microsoft Robotics Studio (1.5)\bin\ " /referencepath:"C:\Microsoft Robotics Studio (1.5)\bin\ "" exited with code 20. wiimoteNxt

Al said...

Evan, have a look at this post. I think they solved the problem you have (a weird system problem)

BTW, did you have similar errors when you built the wiimoteLib project (after running the dssProjectMigration tool) or was it just for the wiimoteNXT service?

Evan said...

ok now i get:
Error 1 Program 'C:\Microsoft Robotics Studio (1.5)\wiimoteNxt\obj\Release\wiimoteNxt.Y2007.M11.exe' does not contain a static 'Main' method suitable for an entry point wiimoteNxt

cybercupido said...

yo tengo un tuto de como controlar el NXT con en labview, q esta mucho mas sencillo y ademas el tutorial esta en espaniol.. visita lopezpaul.net

CybrRyno said...

I keep getting this error when I debug the program-

Error 1 The type or namespace name 'WiimoteLib' could not be found (are you missing a using directive or an assembly reference?)

Thanks
Ryan

Al said...

CybrRyno, did you add the reference as explained? (right click on references and click on add, and choose the WiimoteLib proxy reference)

CybrRyno said...

Al - Yes, we did follow those instructions. Would it matter that the wiimotenxt reference that is loaded is "wiimoteNxt.Y2008.M03.Proxy"?

Thanks

Al said...

Well, the reference you have to load is NOT the wiimotenxt one (which is the proxy for the actual service you are writing - wiimoteNxt) but rather "Wiimote.Y2007.M06.Proxy", which is the Proxy of the wiimoteLib service, i.e. the library that allows you to access wiimote data.

If you don't add that reference, the compiler will not recognize the statement:
using wiimote = WiimoteLib.Proxy;
since the wiimoteLib is not referenced.

I hope that was the problem!

mikek said...

Evan, have a look at this post. I think they solved the problem you have (a weird system problem)

Thanks that solved the problem I was having.

hihihi said...

where do you un zip the download to?

Al said...

hihihi, you should unzip it in the MSRS installation folder

Mohammad said...

Hie, Al,
i get this error while running the program,although the program works,


*** Dssp Operation handler has been marked with the ServiceHandlerAttribute but its operation type is not on the operations port.Method:System.Collections.Generic.Ienumerator`1[Microsoft.Ccr.Core.ITask] DropHandler(Microsoft.Dss.Services.DefaultTarger.Drop)

****
Now if i want to add a code something like
**
drive.RotateDegreesRequest rotateRequest = new drive.RotateDegreesRequest(80, 0.5);
_drivePort.RotateDegrees(rotateRequest);
**

i get no response out of the motor,do i have to cfeate a handler of some sort in the IEnumerator(ITask)..?

PS:new to MSRS n C# :)

Al said...

Hey Mohammad,

The error seems to say that you have a handler for a certain message in your code marked with the [ServiceHandler] attribute (a line right before a function with square brackets around it), but that the handled message type doesn't belong to the operations of the service (the class isn't added to the WiimoteNxtOperations PortSet, defined in the WiimoteNxtTypes.cs file).

There are two possibilities: Normally, the only Service Handler in the code should be the GetHandler, so you either removed 'Get' from the WiimoteNxtOperations Portset, in which case you should add it back; or you added the ServiceHandler attribute to a custom handler, in which case you should remove it (or if you need it you could add the message type to the operations portset).


For the motor problem, it is likely that the robot you're using doesn't implement the RotateDegrees message, so it just doesn't do anything.
Beware that drive.RotateDegrees makes the robot turn a certain angle, and not the motor. To do that, the motors must have encoders, and the service needs to know some other parameters specific to the robot, so this is why the message isn't implemented.

Hope this helps,
Alberto

Mohammad said...

Thanks Al,

But i have 2 things,
first, i downloaded the project file, and ran it as it is, and got the error,
*Dssp....
so i can't actually figure out what went wrong,

and secondly that i have lego mindstorms nxt,and i assume that it has a encoder built in ..so RotateDegrees should work i guess..?

Mohammad said...

And apparently i cant get the RotateDegrees to work for me in the MVPL as well...so any ideas where i may be faltering?

Al said...

Mohammad,
for the Dssp thing are you sure it's an error? Because for what I understand, the program is working (it builds fine), and this should mean there are no errors. If it's a warning then it doesn't matter you can leave it there.


For the rotateDegrees:
I checked the source code for the Nxt drive service, and the V1 services don't implement the message. I think those are used by default in MSRS 1.5/C#. The V2 services (used in MS robotics developer studio 2008 and I think in VPL for 1.5 too) do implement the rotateDegrees message, as long as the config file (samples/config/Lego.Nxt.v2.Drive.Config.xml) has the 'DistanceBetweenWheels' property bigger than 0.0 (meters). This is needed for odometry calculations.
Anyway I suggest moving to MRDS 2008 using the DssMigration Tool the same way I showed it in the tutorial.

Good luck!
Alberto

Luigi said...

Well! Nice work!!

But I got some buid errors and I can't fix it.

Here they are:

Error 1
'WiimoteLib.Proxy.AccelState' does not contain a definition for 'Y' and no extension method 'Y' accepting a first argument of type 'WiimoteLib.Proxy.AccelState' could be found (are you missing a using directive or an assembly reference?)

Error 2
'WiimoteLib.Proxy.AccelState' does not contain a definition for 'X' and no extension method 'X' accepting a first argument of type 'WiimoteLib.Proxy.AccelState' could be found (are you missing a using directive or an assembly reference?)

And another thing... it is possible to use the RCX version of Mindstorms? If yes, what changes do I have to do?
Thank you!

fredflintstone said...

I also received an error message regarding the AccelState does not contain def for Y, nor X.
Any idea?
Thanx,

Al said...

luigi and fredflinstone,

I just updated to the new WiimoteLib library (available here)), and it looks like they changed the AccelState class:

you should now use:
AccelState.Values.X
instead of
AccelState.X
(same thing for Y)

sorry for this inconvenience. I'll change the tutorial as soon as I can, and maybe I'll also update for Microsoft Robotics Developer Studio 2008.

I hope this helps!
Alberto

Pablo said...

hey dude, I have this problem and I don't know what I do wrong:

In the file "wiimoteNxt.cs"
in the final line who say "_drivePort.SetDrivePower(request);"

when I debug the program say "no found NullReferenceException"
what can I do?

Al said...

Hey Pablo,

I'm not sure where that exception comes from, but I actually got the same one exactly. I got it twice in a row, but then it worked! (maybe after a restart? who knows..)
it's weird though...

If it still doesn't work, try to ask on the MSRS forums here, you'll definitely get a better answer!

Good luck!

Machelo said...

hi..!!
well im a student, im 15 years old, im from ecuador, and i'd like to do this as my proyect in high school..!!

but i dont understand a lot, at the end you said that we can download the proyect, is there everything you did there???

i mean that only with that i can execute the proyect??

hope you understand.. hehe

pandaemonium said...

I tried to load your code OR create the service in MSRS 2008 with Visual Studio 9.
I "copied the files from the "wiimoteLib dlls" folder into the /bin/ folder under the MRDS 2008 installation directory" and yet the service is not showing up.
Why?

Wei said...

Hi:
Thanks for posting this project, nice work ~!

I tried out the code, everything seems fine, but at the end when i debug it, I am getting this at the command prompt,




*** Service creation failure most common reasons:
- Service contract identifier in manifest or Create request does not match C
ontract.Identifier
- Service references a different version of runtime assemblies
Additional information can be found in the system debugger log.
[07/16/2009 00:09:16][http://wei-pc:50000/manifestloader/8dff8389-cd88-4df2-817
5-e3a29a29fea4]

do you have any idea what this is? Thank you

Wei said...

Hi:
Thanks for posting this project, nice work ~!

I tried out the code, everything seems fine, but at the end when i debug it, I am getting this at the command prompt,




*** Service creation failure most common reasons:
- Service contract identifier in manifest or Create request does not match C
ontract.Identifier
- Service references a different version of runtime assemblies
Additional information can be found in the system debugger log.
[07/16/2009 00:09:16][http://wei-pc:50000/manifestloader/8dff8389-cd88-4df2-817
5-e3a29a29fea4]

do you have any idea what this is? Thank you

Wei said...

Hi:
Thanks for posting this project, nice work ~!

I tried out the code, everything seems fine, but at the end when i debug it, I am getting this at the command prompt,




*** Service creation failure most common reasons:
- Service contract identifier in manifest or Create request does not match C
ontract.Identifier
- Service references a different version of runtime assemblies
Additional information can be found in the system debugger log.
[07/16/2009 00:09:16][http://wei-pc:50000/manifestloader/8dff8389-cd88-4df2-817
5-e3a29a29fea4]

do you have any idea what this is? Thank you

johnc_gl said...

Hello and sorry bothering you.
I try to make my master degree diploma in Computer Intelligent Systems using MRDS 2008 R2. I saw your work and it's pretty pretty nice. Congrat's. I'm pretty new with the environment. I try to control a Lego NXT Robot or other robot in the simulated environment because i don't have the hardware. Can you provide me some clues?

johnc_gl said...

Hello and sorry bothering you.
I try to make my master degree diploma in Computer Intelligent Systems using MRDS 2008 R2. I saw your work and it's pretty pretty nice. Congrat's. I'm pretty new with the environment. I try to control a Lego NXT Robot or other robot in the simulated environment because i don't have the hardware. Can you provide me some clues?