Sockets for AGS (alpha 1) -- testers required!

[Wyz]

Yes! Sockets for AGS. There are already a variety of socket plugins out there (one or two by me ) for AGS but this one tries to be as general as possible. However it is not yet finished! The reason why: it needs more testing; I have tested it but to test this plugin properly I actually need other people to play around with it.

So that is where the community comes in! I need a couple of developers that already have some understanding of sockets to test it. You can leave all your findings, comments, questions and requests here in this topic. I'll post updates when I have them also here. If you have something to showcase you're also free to post in this topic. Since I don't have documentation (except for the auto-completion hints which you should try) if you're unsure how a functions works or what it returns you can also post it here; it will also give me an indication of where to put the focus when writing the manual.

So I proudly present:

agsSock Alpha 1

Warning: This plugin is not compatible with the old plugins (including mine). If you're already using a sockets plugin in a project make a backup of the old plugin!

So the current state: Some features are not yet implemented or optimized and proper documentation (a manual) is lacking. Other then that it is presumed fully functional.

Some of the characteristics of this plugin (design decisions):

General
The plugin tries to be close to Berkeley sockets (for those who know it) to take advantage of a widely used standard. Yet it will also wrap around certain things to make it more convenient for use in AGS. The biggest change is that it is object orientated.

Object orientated
AGS has been object orientated for the past few versions so let's take advantage of that. Sockets and addresses are objects which also keep internal records. This makes it possible cache information for efficiency reasons. It keeps track of validity and errors for each socket independently. It also has a built-in buffer.

Buffered
Every socket has a buffer for incoming messages meaning that you don't have to worry about read errors except those that are fatal. The buffers and error states are updated in a separate thread.

Threaded
The plugin is threaded: that means that even if AGS is waiting for user input the plugin will still process sockets in the background like reading incoming data or processing errors. It can do this with a single thread that is only active when something actually happens. This makes receiving messages effectively non-blocking.

Non-blocking
The entire plugin tries to be non-blocking. Blocking is where the application will stop and wait till it can complete the action (like waiting till a message was sent so it can receive it or a connection was made). Blocking is not very convenient for use in AGS because that will make the game unresponsive. Instead all methods will return unsuccessfully (but without an error number beware for amusing errors messages) if nothing happened.
There are two exceptions:

• Resolving addresses (converting addresses to string and the other way around)
• Connecting synchronously (default)

However it is possible to work around both in cases where it becomes a problem.

A standard
The plugin is based on a standard but also tries to be one itself. When it's stable my intentions are to also make modules for commonly used communication protocols like HTTP. I'll try to make a proper manual and provide examples for those who want to use it themselves. For now all I have is auto completion hints though.
It also tries to be as portable as possible but for now I will only release windows binaries (it should be posix compatible but I've never tested it).


I hope for now people play around with it and it will find an use in the future. It can also be a stepping stone for those who want to learn more about computer networks. Maybe we'll see the first multi-play adventure game. I'm excited to see all that!

[DoorKnobHandle]

Here is an open-source sample that might help you get started: http://dkh.agser.me/sock.zip (AGS 3.2.1)

To test this out, start Compiled/sock.exe twice, click on "Server" in one instance of the game, click on "Client" in the other. The connection should be set up and you should be able to chat with the other game!

This is not useful in itself but it shows the steps (look at TCP.asc) necessary to get started with this plugin!

EDIT: If anybody wants to update/improve/document this sample, go right ahead please!

[Wyz]

Thanks for the demo program dkh, look pretty good.

I guess I'll also a little example so people get a feeling how things work. This is a multi-client chat server you can use with a telnet client. Simply connect with your client to localhost.

Here it is:

Code: ags

// room script file

Socket *server;

//- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 

struct Client
{
  Socket *sock;
  String buffer; // unprocessed input
};

#define MAX_CLIENTS 10
Client client[MAX_CLIENTS];
int clients = 0;

//------------------------------------------------------------------------------

void AddClient(Socket *sock)
{
  client[clients].sock = sock;
  client[clients].buffer = "";
  clients++;
}

//------------------------------------------------------------------------------

void RemoveClient(int id)
{
  clients--;
  client[id].sock   = client[clients].sock;
  client[id].buffer = client[clients].buffer;
  client[clients].sock   = null;
  client[clients].buffer = null;
}

//------------------------------------------------------------------------------
// Adds input to the client's process buffer

void BufferPush(this Client *, String msg)
{
  this.buffer = this.buffer.Append(msg);
}

//------------------------------------------------------------------------------
// Fetches the next command from the process buffer (if there is one),
// otherwise: returns null.
// Note: commands are terminated by a carriage-return and linefeed character.

String BufferPop(this Client *)
{
  int i = this.buffer.IndexOf("\r\n");
  if (i < 0) return null;
  
  String str = this.buffer.Truncate(i);
  this.buffer = this.buffer.Substring(i + 2, this.buffer.Length - i - 2);
  return str;
}

//------------------------------------------------------------------------------

void SendAll(String msg)
{
  int i = clients;
  while (i)
  {
    i--;
    client[i].sock.Send(msg);
  }
}

//------------------------------------------------------------------------------

function entry() // Enters room after fade-in
{
  SetMultitaskingMode(1);
  server = Socket.CreateTCP();
  
  if (!server.Bind(SockAddr.CreateFromString("telnet://0.0.0.0")))
  {
    Display("Unable to bind to port: %s", server.ErrorString());
    return;
  }
  
  if (!server.Listen())
  {
    Display("Unable to listen: %s", server.ErrorString());
    return;
  }
}

//------------------------------------------------------------------------------

function loop() // Repeatedly execute
{
  Socket *sock;
  
  // Debug text
  if (server == null || !server.Valid)
  {
    lblOutput.Text = String.Format("Disconnected!", server.Local.Address, clients);
    return;
  }
  lblOutput.Text = String.Format("Listening: %s, %d clients", server.Local.Address, clients);
  
  // Accept new connections
  sock = server.Accept();
  if (sock)
  {
    // Client connected
    AddClient(sock);
    sock.Send("Please enter a nickname: ");
  }
  else if (server.LastError)
    Display("Unable to accept connection: %s", server.ErrorString());
  
  // Process messages
  String msg;
  int i = clients;
  while (i)
  {
    i--;
    
    msg = client[i].sock.Recv();
    if (msg != null && msg != "") // Non-empty message arrived
    {
      client[i].BufferPush(msg);
      
      msg = client[i].BufferPop();
      while (msg != null)
      {
        // Incomming message
        if (client[i].sock.Tag == "")
        {
          client[i].sock.Tag = msg;
          SendAll(String.Format("%s entered.\r\n", client[i].sock.Tag));
        }
        else
          SendAll(String.Format("[%s] %s\r\n", client[i].sock.Tag, msg));
        msg = client[i].BufferPop();
      }
    }
    else if (msg != null && msg == "") // Empty message arrived
    {
      // Client closed connection
      SendAll(String.Format("%s left.\r\n", client[i].sock.Tag));
    }
    else if (client[i].sock.LastError) // Trouble arrived
    {
      // Client had an error
      SendAll(String.Format("%s was disconnected: %d\r\n", client[i].sock.ErrorString()));
    }
    
    // Clean up lost clients
    if (!client[i].sock.Valid)
      RemoveClient(i);
  }
}

[dbuske]

I never heard of a socket. Is it a network protocal?

[Construed]

Beautiful work!
I'm a little too far with the old plugin to switch over at the moment, but I will dabble with it soon and see if I can help debug.

[Wyz]

Thanks! In a few weeks when I'm not as busy as right now I will make a few more and more practical examples; I guess that will generally be more useful then the very bare state the plugin is in right now.

I never heard of a socket. Is it a network protocal?

Sockets are the end points of a network protocol; if you view a network protocol as a cable then the sockets are the places where the cable is connected. That way you can tell what the network is connected to and send and receive messages to and from the other side.

[RickJ]

Wyz,

Bless you sir!  I shall take this us next week.

Thanks

[Crimson Wizard]

Works with 3.3.0 beta (dkh's demo script).

I think I will try making a simple multiplayer game with this.

[Crimson Wizard]

Every now and then I am getting an error when starting AGS:

There was an error loading plugin 'agssock.dll'.

Unable to load plugin 'agssock.dll'. It may depend on another DLL that is missing.

After some time (or maybe some actions) it loads fine.


On other topic, is there a way to a) refuse client connection and b) drop previously connected client?

[Construed]

I don't exactly know how this version works but some commands from the irc version:

Code: ags

void Connect(String nick)
{
  if (!IRCConnect(SERVER))
  {
    Display("You failed to connect, Please try again.");
    return;
  }
  
  if (String.IsNullOrEmpty(nick))
    nick = DEFAULT_NICK;

  nickname = nick;

  channel = Room.GetTextProperty("channel");
     
  IRCNick(nick);
  IRCUser(nick, "agsIRC");
  player.Name = nick;
  
  connected = true;
}

Then use:Connect(); or Connect(nick);

And for disconnect:


void Disconnect()
{
  connected = false;
  initiated = false;
  IRCDisconnect();
}
and call the:
Disconnect();

I hope you figure it out my friend, I also want to test this version soon!

[Crimson Wizard]

Hmm, no, this plugin has completely different set of commands, nothing with "IRC".

E: I see how I can shutdown the socket ran as "host", and I see how I can disconnect client being the client, but I do not see how to drop client connection when being the host (i.e. "kick" another player from the game). Or maybe I am missing something.

[Construed]

Ah, yes... Much to advanced for me at the moment.
You should PM WYZ and tell him about this post, Sometimes its hard to notice a topic amongst all the other.
Hope you figure it out bud, I plan to dabble with this too when I get some spare time

[Wyz]

Sorry for my very very late response!

Every now and then I am getting an error when starting AGS:
(...)

I've heard about this error before but have been unable to reproduce it myself. One person claimed it had something to do with steam running in the background but I have no way of confirming this unfortunately. If this happens regularly place let me know.

On other topic, is there a way to a) refuse client connection and b) drop previously connected client?

You can refuse incoming connections but only without knowing what remote host is trying to connect. This mechanism is mainly used to stop accepting any new connections when the server has met its capacity. The way this works is that there is a queue on the listening end that holds new connections. When this queue is full new connections will receive a 'connection refused'. The size of this queue is called backlog and can be specified by the Listen method.
Connections will be left waiting in the queue until you call the Accept method. At that point you grant them a connection and you also get their ip.

So how to refuse certain connections? Well you accept then an then immediately close them optionally sending some error along. So that bring me to part (b) of your question: how to close them?
Accept returns a new Socket object representing this new connection. Call their Close method to close them gracefully (connection closed) or dereference them (set the pointer to null) so that AGS' garbage collector deletes them which closes them in not so a graceful way (connection reset).

I hope that answers your questions and again sorry for my late response.

[Khris]

First of all, amazing plugin!
I also got the "Unable to load plugin 'agssock.dll'." error about an hour ago, couldn't resolve it, restarted my PC and it was gone.

But, I'm more interested in the proper way to read HTTP data. I'm sending a GET request, then simply listen in a while loop until Recv() returns something other than null. The thing is if I run this on my old laptop, I'm not getting the complete reply in one go; right now I'm using two nested loops and a timeout to make sure I'm getting the entire message. I noticed that there's a 0 (zero) after the content; do I have to keep reading lines until I get to the 0? What if my data actually contains \r\n0\r\n? Unlikely but possible, right?
Basically, how do I read the entire content without essentially halting my game for a few seconds? Assuming this is possible, of course.
Or do I have to live with this and just wait for either the zero or a timeout?

[Wyz]

Thanks! Thanks for trying it!

The plugin uses an internal buffer (other then the buffer at system-level) to store incoming data until you use Recv. It processes incoming data as it comes in a separate thread and puts it in the buffer. When the connection was closed an empty string is returned by Recv. Since http servers typically close the connection by default after sending the response you can assume you received everything after the connection has been closed (thus you get a empty string from tRecv). You can force this behaviour by sending the header Connection: close along your request.

Because the plugin is threaded you don't have to do a wait loop and you can put the Recv calls in execute_repeatedly. When it returns null and the error code is 0 it means there has not yet been a response for the server so you can skip processing that frame.

I don't know exactly what the 0 after the content is. When you're using a http 1.1 request the server should return something like:
STATUS\r\nHeader1: ...\r\nHeader2: ...\r\n\r\n<data>
The request should look something like his: (note the extra \r\n at the end)

Code: ags

GET /index.html http/1.1
Host: www.someurl.com
Connection: close

So yeah everything after the first double CRLF should be data if all is well. Let me know if that works.

[DoorKnobHandle]

I also get the additional 0 as well (and an extra b for some reason?):

Code: ags

HTTP/1.1 200 OK
Date: Sat, 01 Feb 2014 16:39:10 GMT
Server: Apache
X-Powered-By: PHP/5.4.21
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html

b
Hello Hans!
0

This is me sending a similar 1.1 request as the one you posted above, Wyz (except it should be forward slash at the end of the first line I believe) to a php script on dkh.agser.me. I was also wondering where those extra characters come from, no idea so far. The php script obviously only does "print 'Hello Hans!'"

[Khris]

I'm getting a 59 before my highscore string.
First I thought it's the length, but the String's length is actually 89. I thought about your b and suddenly the lightbulb came on: it is the length, but in hex

Edit:
Changed some things around, implemented proper parsing.
I'm getting this:

Code: txt

HTTP/1.1 200 OK
Date: Wed, 19 Feb 2014 02:04:49 GMT
Server: Apache
X-Powered-By: PHP/5.4.21
Connection: close
Transfer-Encoding: chunked
Content-Type: text/html

51
1. selmiak: 108[2. Khris: 77[3. Arnjeir: 39[4. Peder Johnsen: 25[5. klotzkopp: 20

Flawless!
Thanks again Wyz, great job!

[Wyz]

Oh I know what that is; the response header gave it away Yes that is chunking, a way for http servers to split up the message body in chunks. No way around it unfortunately and some servers are particularly eager to use it. Nice! Looks like you already found out but for the record you can read here how to parse it. Transfer encoding can be a pain sometimes.

(btw I fixed the slash, that was a typo )

[Khris]

I'm trying to get binary data. Apparently I can't do that using Strings though because those omit 00 chars.
Tried to do it with RecvData() instead and I'm getting nothing, it always returns null. Is this not implemented yet?

Edit: nevermind, just discovered a really stupid typo in my code
Edit: actually, still not working...

[Wyz]

Yes you're right, RecvData has not yet been implemented. It will become available in the next version but for now it doesn't work unfortunately.
Maybe Base64 encode the data for now? I have a base64 decode module you can use if you want.

[Khris]

Ok, that explains it
I got base64 decoding to work after a few hiccups and got AGS to display an image loaded from a server, the implications of that alone...
When I tried a bigger image, agser.me borked though, but it did work from localhost. Now to parse chunked data...
Unfortunately, AGS can't create DynamicSprites from PNGs, but that's another story.

Nice timing btw, I was literally about to hit the sleep button and go to bed when I saw your reply, and I stayed up for a couple of hours to try it immediately

[Khris]

Alright, I'm running into a serious problem.
I'm trying to transfer a bigger image; it's a PCX with a file size of about 45k. What happens is khris.agser.me sends base64 data, about 60k.
I've tried both text/plain and text/base64, as well as HTTP/1.0 and HTTP/1.1. The first option has no effect, but when I use 1.1, the server breaks it up into chunks. So far, no problem there, what happens is I'm only getting about 12k. At this point server.Recv() apparently returns null, because AGS stops reading. I'm positive this isn't an error in my code because reading the same data from localhost works fine and I'm getting the entire 60k.
The error is also not with the remote server, because when I request the document in Firefox, I'm getting the full 60k in text format.

In this vain, if I wanted to read the data asynchronously, is server.Recv() non-blocking? Because if I have a slow connection, how is it ensured that Recv() will always return something as fast as I'm calling it?

[Wyz]

Sorry to have deprived you from sleep. Whoah, you are fast

Yes Recv is non-blocking. When there is nothing in the buffer because it is still under way Recv will return null. Note that this is different from an empty string which would signal that the connection is closed. There are two scenarios that would return null: an error has occurred (LastError > 0) or the call 'would block' which means you have to wait for new data to arrive (LastError == 0).

[Khris]

Thanks, I should've realized that. On the upside, I managed to download a 1.1 MB file and save it to the game dir Unfortunately, decoding base64 takes over a minute with filesizes like that so I'll wait till you implement RecvData().

[Monsieur OUXX]

Is there a macro defined when the plugin is present? That would be very useful to enable or disable some code in the AGS script.

[Wyz]

Not yet but I've added a define for the next version. 

Code: ags

#define AGSSOCK

[Monsieur OUXX]

Not yet but I've added a define for the next version. 

Code: ags

#define AGSSOCK

So when do you release it?

[Wyz]

I can't give an estimation unfortunately.  You can add the flag manually for now I guess.

[Monsieur OUXX]

I can't give an estimation unfortunately.  You can add the flag manually for now I guess.

Sure, will do. Thanks for the answer. Please, it would be cool it it weren't too long -- I'm not really interested in the new features, but that flag is important for our compilation/releasing process, to avoid a lot of human operations and human mistakes.

[Crimson Wizard]

[SCRAPPED] Sorry, I just realized that's not really something that was needed here.

[Crimson Wizard]

May anyone (WyZ?) please elaborate, how do I use asynchronous connection to address?
This is how I begin async connection:

Code: ags

sock.Connect(addr, true);

But how do I know that/when connection was established? Should I check some property over time, or call any function? Recv() maybe?

Secondly, do I understand this right that getting "" from Socket.Recv() is the correct way to know that the remote socket has disconnected? Is there any property that indicates this fact for certain?

[Wyz]

May anyone (WyZ?) please elaborate, how do I use asynchronous connection to address?

Yes it's all undocumented for the time being unfortunately  , I've kept close to the Berkeley sockets for as far as I could. The trick is to call Connect multiple times.
If you call Connect asynchronously (with async == true) one of three things can happen:

• Connect returned false and LastError == 0: The connection is not yet established, you need to call Connect again.
• Connect returned false and LastError != 0: Some error occurred and the connection could not be established. (You could use ErrorString to see what went wrong)
• Connect returned true: The connection is established, the socket is ready to be used.

I believe for this to work correctly you need to use the same SockAddr object for each call. I don't remember if I ever tested this but I advise you to do so.

Secondly, do I understand this right that getting "" from Socket.Recv() is the correct way to know that the remote socket has disconnected? Is there any property that indicates this fact for certain?

Getting "" from Recv is the way to know the remote end closed the connection gracefully: "hey I wan't to stop transmitting". Some clients simply cut the connection which makes it impossible for them to check if everything they sent really came through. However since they are out there you probably also want to deal with clients not closing gracefully. In that case (but also other cases) Recv returns null and the LastError is set to something non-zero. There are error codes that will tell you the connection has been cut specifically ("connection reset by peer") but since error codes are platform specific I don't really want to go into that.

I hope that helps you.

[Crimson Wizard]

I found following issue.
When I call Sock.Connect(addr, true) for the first time, it returns false and LastError = 0. When I call it for the second time, it returns false and LastError is 10056, which is "Socket is already connected".
Another game instance that ran "server mode" detected that connection was made.

Regarding error code and strings, I think they are unusable like this. Not only error code is OS-specific, as you say, but also error string is in OS language and may contain characters unsupported by game font.

[Wyz]

I guess that is either a flaw in my plugin or I remembered incorrectly how this is supposed to be done (it has been a while). I will look into this for sure but for now I guess the work around is to see if you can read or write after Connect returns false. If not either the connection was closed immediately or it was never connected to begin with.

I'm aware that error codes themselves are pretty useless this way and can only be used effectively to see if there were any errors or not. Since there is no standard at all for error codes and some are very platform specific I don't really know how to deal with them other then to just pass them. That way it doesn't stop other people from making platform specific modules to wrap around them. The error string is there to aid the developer in debugging and I wouldn't recommend them to be passed to the end-user. I didn't really think about the localization problem; I guess they are in Cyrillic for you which AGS wouldn't be able to render I assume: I'll see if I can force them to be shown in English.

[Crimson Wizard]

At the moment I am writing error strings into file. This lets me to read them if I set right code page when opening the file.

[Dualnames]

I've open sourced the AGS ceremony files, so perhaps they can be of some help??

[Crimson Wizard]

I've open sourced the AGS ceremony files, so perhaps they can be of some help??

I wasn't aware they use this very plugin; I am going to check them, thank you.

[Crimson Wizard]

Every now and then I am getting an error when starting AGS:

There was an error loading plugin 'agssock.dll'.

Unable to load plugin 'agssock.dll'. It may depend on another DLL that is missing.

After some time (or maybe some actions) it loads fine.

A small update on this problem.
I was working on my project on different computer for few days, and this error never occured to me (although it was reoccuring every now and then before).
The difference is that it has Windows Vista installed (as opposed of Win7) and not a part of any net (nor connected to Internet).

[Wyz]

That's very interesting, thanks for sharing!
When I was working on another plugin not too long ago I had the same error btw. I have strong reasons to believe it has something to do with the compiler I'm using. Unfortunately I'm still completely lost why it happens. For release versions I'm going to use a different compiler I guess; it's a pain however.

[Sledgy]

There're AGS games with agssock?

[Monsieur OUXX]

Every now and then I am getting an error when starting AGS:

There was an error loading plugin 'agssock.dll'.

Unable to load plugin 'agssock.dll'. It may depend on another DLL that is missing.

After some time (or maybe some actions) it loads fine.

A small update on this problem.
I was working on my project on different computer for few days, and this error never occured to me (although it was reoccuring every now and then before).
The difference is that it has Windows Vista installed (as opposed of Win7) and not a part of any net (nor connected to Internet).

I have the same issue. It appears from time to time, then I make it disappear by opening./closing/rebuilding/rebooting/etc. I'm never sure what fixes it. It seems to happen randomly and disappear randomly. I have Win7, I use AGS 3.3.0 RC1.

[Crimson Wizard]

I just wanted to mention, WyZ did not update this forum thread for a long while, but this plugin source is now hosted on github:
https://github.com/FTPlus/AGSsock

And here are recent releases:
https://github.com/FTPlus/AGSsock/releases

For the reference, here's AGS Awards client code that uses this plugin and is basically a big example of a network client:
https://bitbucket.org/agsa/ags-awards-source/src/master/AGS%20Project/

[Wyz]

Yes true, there have been some discussions in other threads about the plug-in though.

The status right now is:
Code-wise, the plug-in is ready for release, but there is no documentation. I looked for some assistance for this, as it is not really my forte, and I'm pretty busy anyways. Eri0o has joined in; and things are moving again, although I'm not sure how fast. If anyone has ideas for tech demos or code examples please share; if its doable, I'll happily crank that out. Note that the AGS Awards ceremony is based on a tech-demo I once made with the plug-in.

When this final stage is done, I'll make a new thread, and put in some fireworks or something.

I hope to add some modules that make use of the plug-in, like a HTTP client for instance.