AC Controller: ESP8266 Telnet Server for logs

Since the start of the project I kept thinking I needed a way to be able to get logs from the system remotely. I originally considered some remote server, maybe use adafruit.io service to push error codes with mqtt, which will allow me easy remote visualization and analysis. I am still considering that option, but as I have already expressed, I hate depending on services provided by thirdparties, so even if I end up taking that path, I decided to add a telnet server to the ESP8266 and send the logs there.

For those not familiar with Telnet, it is just a TCP socket in port 23 where you send ASCII data... super simple... can not get simpler than that... I found this example: https://github.com/esp8266/Arduino/tree/master/libraries/ESP8266WiFi/examples/WiFiTelnetToSerial doing telnet to serial and pretty much reused that.

This is what I did. At the declarations I added:

1. WiFiServer telnet(23);
2. WiFiClient telnetClients;
3. int disconnectedClient = 1;
4. 
   
5. #define DEBUG_LOG_LN(x) { Serial.println(x); if(telnetClients) { telnetClients.println(x); } }
6. #define DEBUG_LOG(x) { Serial.print(x); if(telnetClients) { telnetClients.print(x); } }

Basically for declaring the server (line 1), the client (line 2) and a pair of macros for sending logs to serial only or to serial and telnet if there is a telnet client.

In setup:

1.   telnet.begin();
2.   telnet.setNoDelay(true);
3.   DEBUG_LOG_LN("Telnet server started");  

Start the telnet server (line 1), enable no delay (line 2) which disables the naggle algorithm that would buffer data before pushing it out on the TCP socket.

In loop:

1.   if (telnet.hasClient()) {
2.     if (!telnetClients || !telnetClients.connected()) {
3.       if (telnetClients) {
4.         telnetClients.stop();
5.         DEBUG_LOG_LN("Telnet Client Stop");
6.       }
7.       telnetClients = telnet.available();
8.       DEBUG_LOG_LN("New Telnet client");
9.       telnetClients.flush();  // clear input buffer, else you get strange characters 
10.       disconnectedClient = 0;
11.     }
12.   }
13.   else {
14.     if(!telnetClients.connected()) {
15.       if(disconnectedClient == 0) {
16.         DEBUG_LOG_LN("Client Not connected");
17.         telnetClients.stop();
18.         disconnectedClient = 1;
19.       }
20.     }
21.   }

This will connect to a telnet client if there happens to be one available (lines 1 to 12) and will disconnect the client if the connection gets dropped (lines 14 to 21). With this code, if a client comes in, the current client connection gets dropped (lines 3 to 6).

Lastly, you may notice that I replaced all the Serial.print and Serial.println with DEBUG_LOG and DEBUG_LOG_LN. Those 2 macros are the ones pushing the logs to serial or both to serial and telnet. I will probably extend those in the future to add line number and time.

As with the whole project, the work is being checked in here: https://github.com/desordenado77/esp8266_netatmo_acController

AC Controller: ESP8266 Netatmo Thermostat control

I happen to own a Netatmo thermostat. The company that supplies the gas for my boiler was nice enough to give it to me as a present this winter. The product is an internet controlled thermostat. It is OK, would recommend it to someone looking for that type of product. Personally I hate the fact that I am using a product which will stop working if the company that supplies the cloud service disappears, or if they decide to start charging a monthly fee, but I got it for free, so I am getting to enjoy it without having to think about those issues... hehehe...

Netatmo is also nice enough to give an SDK to be able to access its devices (up to certain limits, of course), but enough for my needs. So, when I started to think about my AC controller, I came up with the idea of using Netatmo thermostat as my remote control and sensor. I first did some investigation of their SDK API using python scripts which I will check in some day (right now they have my personal information hardcoded, so I need to do some clean up). The process to be able to use the SDK goes through creating a new application in your user area in their web page. That gives you some IDs you will need to use later for identification.

Once I manged to get passed the authentication I met my first issue, the temperature is only updated once an hour. That sucks. Not that I was planning on checking the temperature every minute, but at least 4 times an hour. Doing some research on their forum, I found an undocumented API called "syncthermstate" which will force the thermostat to update the cloud with the current sensor values. This API requires "write_thermostat" scope to be enabled during authentication. I also found that once I get an authentication token and a refresh token, I could just use the refresh token to get new authentication tokens at any time, not requiring me to give my username and password again, which simplifies things. So I managed to get a python script with a bunch of https requests talking with the netatmo cloud and getting sensor readings on request from my thermostat.

My ESP8266 board arrived and the time had come to start porting my python script to Arduino. Turns out https is vaguely supported. There is an https example in ESP8266Wifi (does not look like the place for an https client to be for me, but hey, who am I to judge), here: https://github.com/esp8266/Arduino/tree/master/libraries/ESP8266WiFi/examples/HTTPSRequest
And that seems to be the best example. Issue with that example is that everything is hardcoded, but I used that as a building block for my work. Basically you need to write the whole headers and parse the results. To do that I enabled debugging in my python script (which was using "requests" for the http client) by adding the following in the imports:


import logging
try:
    import http.client as http_client
except ImportError:
    # Python 2
    import httplib as http_client

And this in the code:

http_client.HTTPConnection.debuglevel = 1
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)
requests_log = logging.getLogger("requests.packages.urllib3")
requests_log.setLevel(logging.DEBUG)
requests_log.propagate = True

That showed me the headers as being created by the python "requests" and as received from Netatmo, so I recreated the same thing in Arduino.

One thing I noticed when I managed to send the right headers from Arduino to netatmo was that the reply from Netatmo was using chunked encoding. Chunked encoding is a tool from HTTP 1.1 to packetize the responses. Since I had to write the parser for the responses, I decided to force the server not to use chunked encoding by telling it that the ESP8266 is an HTTP 1.0 client instead of 1.1 (chunked transfer encoding is not supported in 1.0).

You may find the code with all of this in here:  https://github.com/desordenado77/esp8266_netatmo_acController

Right now it does not do any control, it is just the building blocks on top of which I will be doing my final application.

Check httpsPostRequest on how am I doing my POST requests. As you can see my parser of the response is very simple and most likely not error proof. I am also not doing any certificate verification for now, which could potentially be susceptible of a man in the middle attack (somebody impersonating Netatmo servers).

In getThermostatData I am synchronizing my thermostat with the cloud and then reading the temperatures. If any of the calls to Netatmo servers fails I attempt to get a new authentication token using my refresh token.

Hope that helps someone... I will continue working on this project, the things to add next are:
- Real temperature check and AC control.
- Add remote logging capabilities
- LED notification
- Physical control with a button (using short press, long press, very long press for different actions)

AC Controller: ESP8266 WiFi Relay

Long time no write... been busy... but I am back...
This June we had a huge rise of temperatures and I suffered my crappy AC thermostat, which does not seem to measure temperature correctly and has no programability. I needed some fix for that. I could have bought a new thermostat, but that would have been very easy. So I decided to get myself an esp8266 board with a relay so that I could switch my AC on an off. I already have a nice wifi thermostat to control my boiler and the company making that one (netatmo) is nice enough to provide an SDK to access the thermostat data, so my plan is to try to use that as input to this ESP8266 board to turn on and off the AC. Here goes my journey to try to achieve such goal...

I first bought the ESP8266 board with a relay pictured above these lines. On ebay. Less than 9 euros delivered. It was my first go at ESP8266, but I had read that you could use the Arduino IDE to program them, so what could go wrong?

I had to solder the 3 pin serial port and the 2 pin boot mode selector, other than that the board was ready to be used. As for setting up the Arduino IDE, everything seems well explained here: https://github.com/esp8266/Arduino

Install Arduino IDE (1.8.3 in my case)
Open Preferences window (File->Preferences).
Enter http://arduino.esp8266.com/stable/package_esp8266com_index.json into Additional Board Manager URLs field. You can add multiple URLs, separating them with commas.
Open Boards Manager from Tools > Board menu and install esp8266 platform (and don't forget to select your ESP8266 board from Tools > Board menu after installation).
Select the Generic ESP8266 board
Select the device used for serial connection (you need a 3V3 232 adapter)
I opened the Blink Sketch and replaced the GPIO used with number 2, which corresponds to the blue LED mounted on the ESP8266 board. Verify, Upload and got a blinking LED... ready to start the real work